Skip to content

HTTP/3 for the plain HTTPS routes, on UDP 443 beside TCP (QUIC step 1) - #16

Open
frahlg wants to merge 1 commit into
mainfrom
feat/quic-h3
Open

frahlg wants to merge 1 commit into
mainfrom
feat/quic-h3

Conversation

@frahlg

@frahlg frahlg commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

QUIC, step 1: the plain HTTPS routes answer over HTTP/3 on UDP 443 too. The watch's one-shot read gets one round trip shorter, and much shorter under loss. The sockets stay on TCP; QUIC for them is step 2, in paired PRs.

What changes

  • internal/quicdoor listens for QUIC on the same address over UDP and hands each connection to the protocol its ALPN named. Today that is only h3, served by quic-go's HTTP/3 server with the same handler and certificate as TCP. It serves /v1/app/once, /v1/now, /healthz and /source. 0-RTT is off. Extended CONNECT, which quic-go always offers, never reaches a route. quic-go logs nothing.
  • Alt-Svc: h3=":443"; ma=86400 goes on TCP responses, but not on a WebSocket upgrade's.
  • The TLS binding holds. Over HTTP/3, r.TLS is the service's own TLS 1.3 session. Its exporter equals the one the client computes on the same QUIC connection (TestTheExporterOverHTTP3IsTheClientsOwn). That is what the box's hello will bind to over QUIC in step 2.
  • ftwcloudd health URL checks /healthz over TCP and HTTP/3. compose.yml maps 443/udp and runs it as the container's health check.
  • Restarts. A QUIC client hears nothing when the process goes, where a TCP client hears from the kernel. ftwcloudd keeps a stateless reset key in quic-reset.key in the state directory, so the next process resets an old connection's next packet (TestARestartResetsTheOldProcesssConnections: about 10 ms with the key; without it the connection still stands after 3 s).
  • Deploy, as files only. deploy/http3.md covers the security group's UDP 443, the order, how to check it and how to turn it off without a release (FTW_CLOUD_H3=false, which ftw-cloud-deploy keeps). bootstrap.sh raises net.core.rmem_max/wmem_max to the 7.5 MB quic-go asks for. I touched nothing in AWS.

Numbers

The watch's one-shot read (TestOnceReadTimes), from the first byte sent to the decrypted answer, in the real handler with a box on its uplink. scripts/netem-once.sh runs it in a Linux container whose loopback carries netem: 20 ms each way, so a 40 ms round trip, and the loss applies to each packet. 200 reads per row. Cold means a new connection, as after the watch slept; warm means a reused one. TCP today offers only HTTP/1.1; HTTP/2 is there for comparison only.

link read p50 ms p95 ms max ms
40 ms HTTP/1.1 cold (today) 138 144 358
40 ms HTTP/2 cold 138 144 157
40 ms HTTP/3 cold 93 98 103
40 ms HTTP/1.1 warm 46 50 140
40 ms HTTP/3 warm 46 50 90
40 ms, 1 % loss HTTP/1.1 cold (today) 138 200 1183
40 ms, 1 % loss HTTP/2 cold 139 394 1230
40 ms, 1 % loss HTTP/3 cold 94 154 288
40 ms, 1 % loss HTTP/1.1 warm 47 52 351
40 ms, 1 % loss HTTP/3 warm 47 51 133
40 ms, 5 % loss HTTP/1.1 cold (today) 141 1169 2225
40 ms, 5 % loss HTTP/2 cold 143 1165 4299
40 ms, 5 % loss HTTP/3 cold 97 297 1154
40 ms, 5 % loss HTTP/1.1 warm 47 318 661
40 ms, 5 % loss HTTP/3 warm 46 130 220

No read failed in any row. A cold read is three round trips over TCP (TCP, TLS, request) and two over QUIC. Under loss, TCP's tail is a lost SYN or handshake segment waiting out a one-second timer.

Apple's stack against this server (macOS 27, URLSession, loopback)

  • URLSession speaks HTTP/3 to quic-go here.
  • With assumesHTTP3Capable, the first request goes over HTTP/3. With UDP refused, it fell back to HTTP/1.1 in the same time as without the flag. I did not measure a network that drops UDP silently.
  • A WebSocket task in a session that already holds an HTTP/3 connection still opened over HTTP/1.1. So Alt-Svc cannot pull the app's socket onto HTTP/3.
  • Alt-Svc alone did not move URLSession to HTTP/3 on a port other than 443, even after the TCP connection was gone. I could not bind UDP 443 locally to try the real port. So the watch's direct path (one-door step 3) should set assumesHTTP3Capable and keep one URLSession. That needs no change here.

Deploy order (Fredrik)

  1. Open UDP 443 in sg-098992ad382ecf94a (command in deploy/http3.md).
  2. Run bootstrap.sh again for the buffer sysctl.
  3. Deploy the image built from this merge.
  4. go run ./cmd/ftwcloudd health https://cloud.ftw.energy/healthz should print HTTP/1.1 ok and HTTP/3.0 ok.

Opening the port first matters. A client that saw Alt-Svc races QUIC against TCP for up to a day, and with UDP shut QUIC always loses that race.

Checks

  • go vet ./..., go test -race ./..., go test -tags ftwclouddev ./..., scripts/check-imports.sh: pass. TestTheSnapshotArrivesFast (appsession, unchanged) once went over its 100 ms on a laptop at load average 50; alone it passes.
  • TestAWatchReadsItsHomeOverHTTP3: a read, token.get and /v1/now over HTTP/3, with the box's bytes graded live. The sockets answer 4xx over HTTP/3.
  • TestAltSvcIsOnPlainResponsesOnly, TestExtendedConnectNeverReachesARoute, TestAnUnknownProtocolIsRefused, TestNoEarlyData.
  • I built the image locally and ran it as in compose (read-only, nonroot, 443:443/udp). ftwcloudd health passed inside the container and from the host through the UDP mapping.

Not here

About 230 lines of service Go (quicdoor, health.go, the flag).

🤖 Generated with Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b25962b97

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +118 to +121
conn, err := ln.Accept(ctx)
if err != nil {
if ctx.Err() != nil || errors.Is(err, quic.ErrServerClosed) {
return nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wait for HTTP/3 shutdown before returning

When ctx is canceled during an active HTTP/3 request, this Accept returns immediately and Serve runs its deferred tr.Close() without waiting for the concurrently running d.h3.Shutdown. In quic-go v0.63.0, Transport.Close abruptly terminates existing connections, so deploys or SIGTERM can cut off in-flight /v1/app/once requests instead of allowing the advertised five-second grace period. Synchronize with the shutdown goroutine before returning from Serve.

Useful? React with 👍 / 👎.

Comment thread cmd/ftwcloudd/main.go
Comment on lines +231 to +233
if err := door.Serve(ctx, udp, tlsConfig); err != nil {
log.Error("HTTP/3 stopped; TCP serves on", "err", err)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Stop the service when the HTTP/3 listener fails

If Door.Serve exits with a fatal UDP or QUIC listener error, this path only logs it while the TCP server keeps emitting the unchanged Alt-Svc header. The inspected deploy/compose.yml health check can mark the container unhealthy, but the deployment documentation explicitly says that nothing restarts on that state, so clients can keep racing a dead HTTP/3 endpoint indefinitely. Propagate this error to terminate the process, or otherwise stop advertising HTTP/3.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T17:24:50.592844Z 8b25962 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

quicdoor serves h3 with the TCP handler and certificate, Alt-Svc points
clients at it, ftwcloudd health checks both, and compose maps 443/udp.
A stateless reset key in the state directory lets the next process reset
an old connection after a restart. Past 200 new connections a second the
door asks for a Retry, a handshake gets 5 s, and a UDP port that will not
bind leaves TCP serving alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant