FTW Cloud over QUIC: the protocol, and the box's uplink racing the WebSocket with 0-RTT (QUIC steps 2–3)

[frahlg]

QUIC, steps 2 and 3: the uplink and the app session over QUIC on UDP 443, beside the WebSocket, with 0-RTT on reconnect. This PR has the protocol and the box's side, with master merged in and the review's fixes (B1 in the text, B2 and S5 on the box, S1, S2, S7). The service is srcfl/ftw-cloud#22, the iPhone srcfl/ftw-app#74, and the registry name srcfl/ftw-webapp#91.

Contract-pair: srcfl/ftw-webapp@claude/registry-cloud-transports

The protocol (docs/protocol-cloud.md, "QUIC")

• One port. ALPN picks the connection: ftw-cloud-box/1 for the uplink, ftw-cloud-app/1 for the app session, h3 for the plain routes. The certificate is the same. TLS 1.3 inside QUIC carries the exporter, and the box binds hello to it as over TCP.
• The same messages, new framing. The client opens bidirectional stream 0 and sends the target first (/v1/box, /v1/app?home=). After that, each WebSocket message goes on the stream with its length as a QUIC varint in front.
• Bulk on its own stream. A tun that carries a lane 1 frame of the phone's session (snapshots, history) goes on one unidirectional stream each way. A lost bulk packet then never holds up a command, its answer or a tick.
• The service offers only the ALPNs it serves. With the sessions off, a client's handshake for them fails at once and it takes the WebSocket.
• Datagrams only where the newest wins. hb, and live without sources, go as QUIC DATAGRAMs. The freshness rules already show a lost one as a gap (hb.liveSeq), never as a live value. A live that carries its table goes on the stream, and a receiver that already holds a newer live still takes the late table when its srcRev is above the one it holds, and nothing else: Freshness case 17, which cloudwire.View runs.
• Order (S1). One Noise counter covers the control stream, the bulk stream and datagrams, so a bulk burst can overtake a control message by a few hundred. appwire.ReplayWindow is now 1024 (16 words), with feat(appuplink): send each reading through FTW Cloud as it lands #1649's note on paced bulk kept. A receiver drops a datagram outside the window; on a stream the same ends the session.
• Tunnel traffic is never a datagram. Lane 0 carries commands, answers and incremental deltas, which must arrive. Two paths for lane 0 would also show the service which frames are commands. So lane 0 stays on the control stream, where QUIC already keeps it apart from bulk.
• Close codes are the QUIC application error, with the same numbers as today (4403, 4408 and the rest).
• Liveness. The client sends a QUIC PING after 15 s of silence, which holds a NAT's UDP mapping; 45 s without anything ends the connection.
• Racing the WebSocket. QUIC starts first, and the WebSocket follows after 250 ms. The first to open wins, and that winner goes first on the same network for a day.
• A handshake proves less than a session (B2, S5). A QUIC connection that ends before the first session message (challenge for a box, message 2 for a phone), or that the path drops within 60 s of it (cloudwire.QUICProvenAfter), demotes QUIC on that network for a day: the WebSocket goes first, and no QUIC win moves QUIC back meanwhile. Closes the service sends on purpose, from 1000 up, and stateless resets do not count.
• Early data (B1). Only the target and Noise message 1 (with now.tun) go in it, so no cmd or api.req ever can. A phone's session waits for the QUIC handshake, the first copy of an early flight too, so a flight the network dropped and replays later replaces nothing and spends nothing. A box sends only its target early, and its challenge may come back before the handshake ends. Tickets allow early data for a day. See The uplink and the app session over QUIC, with 0-RTT a replay cannot use (QUIC steps 2–3) ftw-cloud#22's TestADroppedEarlyFlightReplayedLaterChangesNothing.
• Before message 2, the service sends a phone only control-stream messages. quic-go packs datagrams ahead of stream data, so without that rule an hb reached a phone before message 2. That was one failed reconnect in 30 under netem.

go/protocol/cloudwire/quic.go holds the ALPNs, timings, varint framing and the datagram rule. The transport names come from the registry through gencontract (S7: protocol.CloudTransports). testdata/cloud-quic-vectors.json holds the bytes, and the service and the iPhone replay them.

The box (go/internal/cloudlink)

• race.go races dialQUIC against today's WebSocket, both to the shard endpoint() names. Each network, named by the box's local address towards the service, remembers its winner for a day. quicFailed and demote carry out the rule above (TestAQUICSessionThatEndsBeforeHelloOKDemotesQUIC, TestWhatCountsAsQUICFailing).
• quic.go is the uplink over QUIC behind the WebSocket's method set (transport.go), so serve, read and the writers barely change:
  • hb and a table-less live go as datagrams;
  • bulk tun goes on the bulk stream;
  • the service's close codes read back as *websocket.CloseError, so the shard redirect's codes work over both.
• A reconnect resumes with the service's ticket and sends the target as early data, so the challenge comes back in the first flight. That is one round trip less before hello. TLS over TCP and QUIC keep a ticket cache each.
• GET /api/cloud gains transport: quic or websocket.
• Tests that name their own Endpoint get the WebSocket alone unless they set Options.QUIC. So every existing test runs as before.
• quic-go v0.63.0 is new in Core.

Registry copy: cloud_transports: [quic, websocket] from srcfl/ftw-webapp#91.

Merge order: this PR first, then the service with its go/protocol pin moved to the merge commit, then the iPhone. Nothing breaks if the service ships before the box: a box without QUIC uses the WebSocket, and the service keeps both.

Numbers: today's WebSocket against QUIC

A real ftwcloudd, a box running Core's own cloudlink, appuplink and command handler, and a test phone in Go (go/protocol and quicdoor.Dial) that pairs with it and speaks the app protocol inside the tunnel. The WebSocket rows are today's path on both legs; the QUIC rows are QUIC on both. Everything runs in one Linux container whose loopback carries netem: 20 ms each way, so every leg has a 40 ms round trip, and the loss applies to each packet on both legs. The readable copy is on. 30 connects and 30 taps per row, 90 s of frames.

The connect rows come from a run with the review's fixes in, where a phone's session waits for the QUIC handshake. The other rows come from the run before the fixes, which do not touch a session once it is open.

link	what	WebSocket p50 / p95 ms	QUIC p50 / p95 ms
40 ms	cold connect → first live number	326 / 333	236 / 248
40 ms	reconnect → first live number	326 / 335	235 / 247
40 ms	tap → ack	101 / 107	97 / 109
40 ms	tunnel frame, box → phone	49 / 55	52 / 58
40 ms	copy live, box → phone	52 / 57 (89 of 90 arrived)	54 / 60 (89 of 90)
40 ms	network switch → live again (Go test phone)	355 (new session)	46 (same session, 0 ticks missed)
40 ms, 1 % loss	cold connect → first live number	326 / 608	260 / 401
40 ms, 1 % loss	reconnect → first live number	327 / 419	265 / 383
40 ms, 1 % loss	tap → ack	104 / 168	101 / 110
40 ms, 1 % loss	tunnel frame, box → phone	50 / 57	52 / 58
40 ms, 1 % loss	copy live, box → phone	51 / 59 (90 of 90)	52 / 59 (88 of 90)
40 ms, 1 % loss	network switch → live again (Go test phone)	375 (new session)	47 (same session, 0 ticks missed)
40 ms, 5 % loss	cold connect → first live number	610 / 2057	380 / 568
40 ms, 5 % loss	reconnect → first live number	434 / 1190	385 / 585
40 ms, 5 % loss	tap → ack	104 / 360	104 / 202
40 ms, 5 % loss	tunnel frame, box → phone	52 / 308	52 / 149
40 ms, 5 % loss	copy live, box → phone	53 / 309 (90 of 90)	52 / 57 (82 of 90)
40 ms, 5 % loss	network switch → live again (Go test phone)	689 (new session)	50 (same session, 0 ticks missed)

What it says: QUIC saves round trips where a connection starts, and it keeps loss on one stream from holding up the others. Cold, it is two round trips less: the TCP handshake and the WebSocket upgrade. A phone's reconnect now costs what a cold connect does, since its session waits for the handshake; 0-RTT still saves a box a round trip, whose challenge comes back in the first flight. Under loss the tails shrink everywhere, and a network switch costs no new session. With no loss, a frame or a tap takes one round trip either way.

The copy's live at 5 % loss is not a clean win. QUIC sends it as a datagram, never again, and lost 8 of the 90; the WebSocket lost none but brought its late ones up to 309 ms after. A lost one shows as a gap in the freshness rules (hb.liveSeq), never as a live value. Neither row measures the age of the newest value at the phone, which is what a person sees.

• Cold connect → first live number: a new connection with no TLS ticket, through the outer Noise handshake (with the box session's message 1 in now.tun), the box session's handshake and hello with its subscription, to the snapshot.
• Reconnect: the same phone again, with the service's ticket. Over QUIC it may send the target and message 1 early, but the session opens only once the handshake is done; the WebSocket resumes TLS.
• Tap → ack: a mode change through the tunnel to the box's cmd.ack.
• Tunnel frame: a lane 0 tick, from the moment the box sent it to its arrival at the phone, on the control stream either way.
• Copy live: the readable copy's live, from the box's wallMs to the phone. Over QUIC it goes as a datagram; the counts say how many of the 90 the box sent arrived.
• Network switch → live again: the Go test phone moves its QUIC connection to a new socket (a new source address, as Wi-Fi to cellular) and the session goes on; the time is the path probe and switch. The WebSocket has to dial a new session; the time is from the switch to its first live number. The iPhone's own migration is not measured.

On the phone: FTWKit in the iOS simulator

LiveQUICTests (in srcfl/ftw-app#74) runs the app's own stack (AppModel, PairModel, CloudCarrier, the tunnel and the session) against a real ftwcloudd and a real Core box with the relay unreachable. It runs in an iPhone 17 simulator. The service and the box run in a container whose netem shapes only the phone's leg, 20 ms each way plus OrbStack's own few ms (ping 56–100 ms); the box reaches the cloud unshaped. The readable copy is off. Each row is one pairing, 30 taps and 6 returns from the background.

phone's link	transport	scan → streaming ms	tap → ack p50 / p95 ms	background → streaming p50 / max ms
40 ms	WebSocket	308	52 / 61	274 / 295
40 ms	QUIC	195	54 / 59	165 / 193
40 ms	race (QUIC won)	167	53 / 59	182 / 187
40 ms, 1 % loss	WebSocket	292	53 / 60	277 / 290
40 ms, 1 % loss	QUIC	171	53 / 59	181 / 321
40 ms, 1 % loss	race (QUIC won)	191	54 / 60	184 / 194
40 ms, 5 % loss	WebSocket	303	55 / 254	286 / 785
40 ms, 5 % loss	QUIC	199	55 / 165	184 / 479
40 ms, 5 % loss	race (QUIC won)	191	53 / 154	191 / 382

The Mac's own swift test gave the same at 40 ms: 305 against 180 ms from scan to stream, and 269 against 184 ms back from the background. In an earlier 5 % run the race gave one session to the WebSocket, because one lost QUIC packet made QUIC miss its 250 ms lead. A QUIC dial that loses now goes on to learn, so that one loss no longer keeps QUIC second for a day.

Keeping an idle box linked

The box uplink stayed idle for 180 s with the readable copy off, at 40 ms, counted packet by packet on the loopback (IP bytes):

transport	what goes every 15 s	bytes per day
WebSocket	service's ping, box's pong, an ACK: 3 packets	1.2 MB
QUIC, once path MTU discovery is done	box's PING (53 B), service's ACK (60 B)	0.65 MB

In the first two minutes, both ends also send path MTU probes of about 1.4 KB beside each keepalive. Over the 180 s window that came to 10 KB, a rate of 4.9 MB a day that stops once discovery ends. A PING every 15 s keeps a 30 s NAT mapping open.

Not checked

• A real phone on LTE, and a real Wi-Fi to cellular move. The switch here is quic-go's own path migration in the Go test phone, which proves the service follows the path. Apple's side of it is untested.
• 0-RTT from the iPhone. Network.framework did not resume TLS against quic-go, with a pinned root or with the root in the simulator's trust store. A phone gains nothing from it now: its session waits for the handshake.
• The age of the newest value at the phone, the cost of a real cellular path, and Apple's QUIC on watchOS.

The rig is not in any repository: the box as a test binary in Core's appuplink, a phone in Go over go/protocol and quicdoor.Dial, run.sh, and netem in an Alpine container on OrbStack. Landing it as a check is a follow-up.

Checks

• go/protocol (go test ./...), and cloudlink, appuplink, appproto in Core: pass after the merges. The window test compares 3,000 shuffled frames against a plain record and fails when the carry between words is removed.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-15T19:31:51.021752Z	180f930	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 180f930b99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve the hostname for QUIC certificate verification

For the production endpoint, addr is resolved to a *net.UDPAddr before this call while tlsConf.ServerName remains empty. quic.Transport.Dial therefore infers the TLS server name from the resolved numeric IP, so the certificate for cloud.ftw.energy fails hostname verification and every production QUIC attempt falls back to WebSocket. The tests miss this because the httptest certificate is valid for its loopback IP; set ServerName from the WebSocket endpoint's hostname before dialing, including when QUICAddr overrides the route.

Useful? React with 👍 / 👎.