Repository navigation
feat: a seam for shards, a quicker hot path, metrics, deploys without a storm, and a load harness - #17
Conversation
… pooled registry, and push workers - internal/shard: which instance holds a home. 256 buckets plus single moved homes, from a JSON file; with none, one shard holds every home. - uplink: a box of another shard hears its host (4421) after hello and before the registry is touched. A home's rate limits live with the home, so a redial no longer refills its tunnel. A box that redials faster than 10 then one per 6 s is turned away (4429). Drain closes every box with 4503 and a spread and starts no box.unreachable. A failed write closes the socket. Restore checks rosters side by side and takes only this shard's homes. - registry: one writer and four readers in WAL, synchronous(NORMAL); a box that reconnects writes nothing in Claim. - push: four alert workers and one widget worker. Alerts are never dropped silently and go again on 429, 5xx or a network error, up to four tries within their expiry. Widget pushes merge per home and go at 20 homes a second, so tomorrow's prices are a stream, not a burst. Flush waits for queued alerts. - internal/metrics: counters, gauges and latency histograms with no home in them, in Prometheus text and a minute's CloudWatch EMF line. - go/protocol pinned at srcfl/ftw feat/cloud-shard-redirect (d74e8d2c). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on loopback, and a load harness - /v1/app closes with 4421 and the host before message 1 is read; /v1/app/once and /v1/now answer 421 with Ftw-Shard. An extension token is now <home>.<secret>, so any instance can route it; a token whose home is not its row's is refused, and one without a home still reads. - SIGTERM: stop accepting, close every box and phone with 4503 and a spread of sockets / -restart-rate (1 s to 60 s), flush queued alerts for up to 3 s, exit. A drained box starts no box.unreachable. - Envelopes and tun carry their arrival time to the phone's writer, so the forward histograms include queueing; drops are counted by reason. - -shards loads the directory; -metrics-listen serves /metrics on loopback inside the container, and `ftwcloudd metrics` prints it; one EMF line a minute goes to stdout for CloudWatch. - cmd/ftwload: N boxes and M phones per box against a local ftwcloudd it starts itself, with live at 1 Hz, tun echoes, reconnect storms and a SIGTERM, and the child's CPU and memory. Exits by itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cloud-scale's shard seam (#17) names shards by string, so the subject carries the shard's name, not a number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t waits for the phones' closes - A box's ping and a home's watch renewal are timers, not goroutines: 10,027 goroutines for 2,000 homes with a phone each became 6,027, and RSS fell 13 %. - Queue entries carry their arrival as a monotonic duration (8 bytes, not a 24-byte time.Time); write buffers come from a pool while a socket is idle. - The drain waits up to 2 s for every app session to write its 4503: in a 3,000-phone run, 2,550 phones got 1006 before. - /metrics has the Go heap, the runtime's memory and goroutines. - ftwload reads the service's own latencies and heap from /metrics, its own CPU, and a Linux child's figures from /proc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cloud-scale's shard seam (#17) names shards by string, so the subject carries the shard's name, not a number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and scaling rule in deploy/README.md - `ftwcloudd move` copies the registry rows of the homes a directory gives one shard, and with -drop forgets them here. A merge never makes a row older: a roster only with a higher rosterSeq, other rows only where they are missing, and a tombstone deletes its home. The same command exports and merges. - The spread after 4503 is at least 5 s (-restart-spread-min), about how long the next container takes to start. - compose: GOMEMLIMIT 1400MiB and a 15 s stop grace period for the drain. - deploy/README.md: Deploys (the drain, step by step, and blue/green behind an NLB), Watching it (/metrics, the minute's EMF line), Load (the runs, the t4g.small estimate, when to add a shard) and Shards (the directory and how to move homes). - CLAUDE.md: the shard, counts and drain invariants, and the line budget per part. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drain sets draining before it takes the list of boxes; a box that was between its check and its registration then sees draining under the same lock and is closed with the spread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Four phones fill a stuck box's socket while a tun for another home goes through ten times, each well under 200 ms; the test fails if the stuck box took every write, so it cannot pass for the wrong reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cloud-scale's shard seam (#17) names shards by string, so the subject carries the shard's name, not a number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ies, durable access writes) - Drain closes boxes side by side without the write lock, within 2 s; a box whose socket was stuck held the drain for 14.5 s before. The phones' drain starts first. - push.Resume starts no box.unreachable for a home another shard holds, so a move's restarts raise no false alarms. - Flush counts alerts waiting to go again, and a worker counts as busy from before its pop, so the exit log says how many alerts are lost. - A suspension, a purge, a dropped or pruned token and a roster with a higher number are synced to disk on the one writer; a power cut no longer undoes them. Renewals and what heals itself stay on NORMAL. - deploy/README: the move's ten-minute window, what a merge cannot delete, and the drain's side-by-side closes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ests prove the same on Linux Linux buffers more than the tun limits let through, so a box that stops reading never filled its socket in CI and the test could not tell. A write that hangs while it holds the write lock is what a full socket does, on every platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5acb77865e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| defer func() { _ = tx.Rollback() }() | ||
| for _, t := range moveTables { | ||
| q := fmt.Sprintf(`INSERT OR IGNORE INTO dst.%s (%s) SELECT %s FROM main.%s WHERE home IN %s`, t, columns[t], columns[t], t, in) |
There was a problem hiding this comment.
Upsert changed child rows during shard merge
During the documented two-pass shard move, a child row can change after the first export but before the final -drop pass. This blanket INSERT OR IGNORE then preserves the stale destination row: for example, a replacement extension token conflicts on (home, device_id), so the newly issued token is discarded while the old token remains, and an updated suspension can similarly retain the wrong epoch and fail to suspend the current device. Because the final pass subsequently deletes the source rows, these updates are permanently lost; tables with mutable rows need table-specific conflict handling that retains the newer value.
Useful? React with 👍 / 👎.
| job.regs = again | ||
| s.retrying.Add(1) | ||
| s.after(wait, func() { | ||
| s.queueAlert(job) |
There was a problem hiding this comment.
Revalidate recipients before retrying an alert
When an initial APNs attempt gets a network error, 429, or 5xx, the retry preserves the original PushReg objects and sends them again without consulting the registry or suspension state. If the device is suspended, removed by a new roster, or unregisters its APNs token during the 2/4/8-second retry window, the later attempt can still deliver the alert to that revoked recipient. Re-filter the failed registrations against current registry and suspension state before every retry.
Useful? React with 👍 / 👎.
| if c.u.draining.Load() { | ||
| reason, _ := c.u.restartReason.Load().(string) | ||
| c.closeWith(cloudwire.CloseRestart, reason) | ||
| return nil |
There was a problem hiding this comment.
Close boxes that are still handshaking during drain
This drain check is reached only after the box supplies its hello, but a WebSocket can remain blocked in the preceding handshake read for up to 15 seconds. Such a connection has not yet been added to u.conns, so Drain cannot close it, and upgraded WebSockets are hijacked connections that http.Server.Shutdown does not manage; the main shutdown path can therefore exit after its short drain while this socket receives an abnormal close rather than 4503. Track and close handshaking sockets, or make the drain signal interrupt this read.
Useful? React with 👍 / 👎.
…drain, durable writes, no move tool) - box.unreachable is recorded as told only once a phone took it; an alert lost to a failure or an exit goes again after the restart. Alerts go again until they expire, after a wait that doubles from 2 s to 5 min with jitter, and each retry goes only to devices that may still get it. A registry read error is retried too. - A breaker holds pushes back for 30 s after ten failures at Apple in a row. Three widget workers share the rate; a failed widgets push goes once more. - internal/wsclose: every close the service sends (4421, 4503, refusals) waits for the client's answer, so the client reads the code and not a reset; a phone that writes message 1 at once reads 4421. The drain reaches every socket from its upgrade on, handshakes included (Codex), and every part that holds sockets is a drainer in main. - The spread's floor is 15 s, longer than the port stays closed while compose starts the next container. - Push registrations, unregistrations, tokens and the away rows are synced to disk. A roster prune lifts a suspension only at the epoch it read, so a suspension made meanwhile stays. - Directory hosts must sit under cloudwire.DefaultHost. - `ftwcloudd move` and the move runbook are gone until a second shard exists. deploy/README and CLAUDE.md say that /mcp and /oauth stay on the default host and forward to the home's shard once there are two. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main.go: keep both the keys/MCP/OAuth flags and the shard, metrics and restart-spread flags; push keeps its own context past the signal, auth and agents stop with the signal. go/protocol pinned to Core master f024251. The pacing test reads the door's drops through uplink.TunDropped, now that #17 counts drops in the metrics set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#19, #20, #21) The MCP stack as one change: ftwkeysd, the one part that can open an agent's session (#18); OAuth for assistants with the owner's approval at home (#19); reads at /mcp (#20); control within the owner's grant, and requests that wait for the owner (#21). Reviewed in two passes; merged with main after #17 (scale), go/protocol pinned to srcfl/ftw master f024251. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paired with srcfl/ftw#1645, which adds the wire (
CloseWrongShard4421,CloseRestart4503,StatusWrongShard421 withFtw-Shard,ShardHostOK,CapShard) and makes Core follow it.go.modpinsgo/protocolat that branch; the pin moves to the merge commit once #1645 lands.Why
FTW Cloud now carries every phone's session with its box, so it must stay fast as homes grow, and grow later without a client release. Three limits came first at an estimated 1k homes: one APNs worker with a queue that dropped, one SQLite connection, and reads under the home lock. A deploy also closed every socket at once, so every box redialled in the same half second.
What
1. A seam for shards that needs no client release
internal/shard: each home lives on one shard. The directory is 256 buckets by a hash of the home id, plus single moved homes, read from-shards. Without a file, one shard holds every home and nothing changes./v1/box: 4421 after hello, before the registry is touched;/v1/app: 4421 before message 1 is read;/v1/app/onceand/v1/now: 421 withFtw-Shard.<home>.<secret>), so any instance can route one without a lookup. No app reads/v1/nowyet, so nothing breaks. A token without a home still reads./mcpand/oauthstay oncloud.ftw.energyand forward to the home's shard once there are two, because other parties' MCP and OAuth clients follow no 4421 or 421.2. The hot path
synchronous(NORMAL), and a box that reconnects writes nothing inClaim.box.unreachableis recorded as told only once a phone took it, so one lost to a failure or an exit goes again after the restart.TestAStuckBoxDoesNotHoldUpAnotherHomehangs four writes to one box while they hold its write lock, as a full socket does, and another home's tun goes through ten times, each under 200 ms.TestAStuckBoxDoesNotHoldUpTheDraindrains past such a box within a second.3. Visibility
/metricslistens on127.0.0.1:9464inside the container; read it withdocker exec … ftwcloudd metrics. It shows:4. Deploys without a storm
-restart-rate500 a second, from 15 s to 60 s. The floor is longer than the port stays closed, about 4 to 8 s, while compose starts the next container;box.unreachable.internal/wsclose), so the client reads the code rather than a reset. A phone that writes message 1 at once still reads 4421.drainerincmd/ftwcloudd, so an agent's sessions are closed and counted in the spread once that part adds itself.BenchmarkRestore10k), about 1.2 s on the host by the estimate below.compose.ymladdsGOMEMLIMIT=1400MiBand a 15 s stop grace period.deploy/README.mdgives the drain step by step, and blue/green behind an NLB for later.5. A load harness:
cmd/ftwloadftwclouddand drives N boxes and M phones per box: live at 1 Hz, a tun command every second echoed by the box, reconnect storms, and a SIGTERM./metrics, plus the service's CPU and memory.-max, and leaves a watchdog in case the harness dies.Numbers
The service ran with
GOMAXPROCS=2on an Apple M5, in a Linux container unless marked. Each run lasted under two minutes.live; a real home also holdsprices,plan,lp,hbandhealth, about 10 MiB more per 1,000 homes.On a t4g.small (unmeasured: an estimate, not run on the host). A Graviton2 core does this work at roughly a third of an M5 core's speed, so I take 3.5× the CPU and the same memory: about 30 % of a vCPU and 70 MiB per 1,000 boxes at one message a second. The first limit is CPU:
GOMEMLIMIT1,400 MiBAdd a shard when, over a busy day, any of these holds:
Between about 1,300 and 4,000 boxes, a non-burstable instance with two vCPUs is the cheaper first step and needs no shard.
Review
A review (Sonnet) found seven things, fixed here and in #1645:
push.Resumearmedbox.unreachablefor homes another shard holds. It now skips them.Flushcould miss one between pop and send. Both are counted now, and the log says how many are lost.synchronous(NORMAL)could undo access changes. A power cut could lose a suspension, a purge, a dropped token or a roster that removes a device. Those writes are now synced on the one writer connection; renewals and everything that heals itself stay on NORMAL.A timeout after Apple took an alert can repeat it. For events the phone's extension drops a repeated
(boot, seq), as the protocol says.Owner's review
Fixed in af24782:
box.unreachablecould be lost for good. It is now told only once a phone took it, and alerts retry until they expire.ftwcloudd moveand the runbook are gone; the redirect seam stays.prices,planandlpin the heap floor.cloudwire.DefaultHost.TestAPruneDoesNotLiftASuspensionMadeSinceItRead).TestTheDrainReachesABoxInItsHandshake). Its P1 on the merge went with the move tool.Laptop rules
go testand a simulator, and the numbers above come from quiet runs.pgrepfound nothing left.Size
The service grows from 5,009 to 6,904 lines of Go without tests:
internal/pushinternal/metrics(new)internal/uplinkinternal/shard(new)cmd/ftwclouddinternal/registryinternal/wsclose(new)cmd/ftwloadis a tool and not part of the service (1,938 lines). CLAUDE.md has the table.Overlap
cmd/ftwcloudd/main.go,internal/server/server.go,deploy/compose.ymlanddeploy/README.md. It has right of way; whichever PR merges second resolves the conflicts.Not here
awslogsdriver and two IAM permissions, which the owner adds.Tests
go vet ./... && go test -race ./...,go test -tags ftwclouddev ./...,bash scripts/check-imports.sh.box.unreachabletold only once taken, retries to devices that may still get them, the breaker, a widget retry, an epoch-safe prune, and the spread.🤖 Generated with Claude Code