Repository navigation
feat(cloud): follow FTW Cloud to the shard that holds the home, and spread the redial after a restart - #1645
Merged
Merged
Conversation
…d spread the redial after a restart The wire for sharding FTW Cloud and for deploys without a reconnect storm: - cloudwire: CloseWrongShard 4421 (reason: the host), CloseRestart 4503 (reason: spread in ms), StatusWrongShard 421 with Ftw-Shard for the HTTP routes, CapShard, ShardHostOK (the host must be cloud.ftw.energy or one label under it), RestartSpreadMs. - cloudlink: the box follows 4421 once, keeps the host in its link record, forgets it after a second redirect in a row or three failed dials, and redials within the spread after 4503 without growing its backoff. - docs/protocol-cloud.md: "Shards and restarts", the close codes, the extension token's home prefix and the numbers. Paired with srcfl/ftw-cloud (feat/scale). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…host must sit under the endpoint - Forgetting the kept host resets the hop count, so the default host's next redirect is followed at once instead of costing another backoff (up to 8 s). The test fails without the fix. - endpoint() dials a kept host only if it is the endpoint's own host or one label under it, so a host kept under an earlier build's endpoint is not dialled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frahlg
marked this pull request as ready for review
October 15, 2026 18:20
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
… box forgets its shard, and shard vectors for every client From the owner's review of #1645: - RestartSpreadMs is 1 s to 60 s; a reason of 0 no longer gives a tight loop. - A 4503 before hello_ok is a failed attempt: the backoff grows instead of resetting, so a service that turns every box away cannot hold them in a loop. - A kept host is forgotten after 3 failed dials over at least 30 s (ShardHostFailWindowMs), which outlasts a shard's restart; three failures took about 3.5 s before. - testdata/cloud-shard-vectors.json: which hosts a client may follow (no port, userinfo, uppercase, trailing dot or missing dot) and how it reads a spread, for Swift, Kotlin and TypeScript to replay. - docs: a command in flight at a drain is unknown, not failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on the default host - A kept host is forgotten after 3 failed dials over at least 30 s; the spread is 1 s to 60 s; a 4503 before hello_ok or message 2 grows the backoff; cloud-shard-vectors.json is named with the other vectors. - A command in flight at a drain is unknown to the phone, not failed. - /mcp and /oauth stay on cloud.ftw.energy, which forwards to the home's shard once there is a second one: MCP and OAuth clients follow no 4421 or 421. - Moving homes needs a tool that comes with the second shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Paired with srcfl/ftw-cloud#17 (branch
feat/scale), which pinsgo/protocolat this branch.Why
FTW Cloud must be able to grow past one instance without a client release, and a deploy must not make every box and phone redial in the same half second. Today a box backs off 0 to 0.5 s after any close, so a restart is a reconnect storm, and there is no way to tell a box that another instance holds its home.
What
Wire (
go/protocol/cloudwire/shard.go,docs/protocol-cloud.md"Shards and restarts")CloseWrongShard4421 on/v1/boxand/v1/app. The close reason is the host that holds the home.StatusWrongShard421 with the host inFtw-Shard, for/v1/app/onceand/v1/now.CloseRestart4503. The reason is a spread in milliseconds. The client redials at a random moment within it, and the close does not count as a failure.ShardHostOK: a client follows only tocloud.ftw.energyor one label under it.CapShard: a box that follows listsshardinhello.caps.<home>.<secret>). No app reads/v1/nowyet, so nothing breaks.Core (
go/internal/cloudlink)cloud.ftw.energyafter backoff.hello_okagainst the service keys it was built with.Tests
go test -race ./internal/cloudlink/ ./internal/appuplink/, andgo test ./cloudwire/ingo/protocol.Review
A review (Sonnet) of both PRs found two things on this side, fixed in 4969254:
TestAfterForgettingAHostTheNextRedirectIsFollowedfails without the fix.endpoint()now uses a kept host only if it is the endpoint's own host or one label under it.Owner's review
Fixed in 26426a6 and 7f40a0e:
RestartSpreadMsreturns 1 s to 60 s, so a reason of 0 no longer makes a tight loop.hello_okcounts as a failed attempt. The backoff grows instead of resetting (TestARestartBeforeHelloOKKeepsTheBackoff).ShardHostFailWindowMs). Before, three failures took about 3.5 s, so a 4 s restart sent boxes to the default host (TestAShardRestartDoesNotMakeTheBoxForgetItsHost).testdata/cloud-shard-vectors.jsonis shared by every client. Swift, Kotlin and TypeScript replay it: which hosts to follow (no port, userinfo, uppercase, trailing dot or missing dot; 29 cases) and how to read a spread (12 cases)./mcpand/oauthstay oncloud.ftw.energy, which forwards to the home's shard once there is a second shard;Not here
shardand the apps follow 4421.🤖 Generated with Claude Code