Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/cloud-quic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"ftw": patch
---

The FTW Cloud uplink tries QUIC on UDP 443 first and falls back to the WebSocket when UDP does not get through within 250 ms, remembering per network which one opened. A QUIC session that ends before `hello_ok`, or that the path drops within a minute of it, puts the WebSocket first on that network for a day. Over QUIC, `hb` and a `live` without its source table go as datagrams, and the tunnel's bulk on a stream of its own, so history never holds up a command. Noise sessions accept frames up to 1024 out of order, since bulk can overtake lane 0. `GET /api/cloud` says which transport carries the link. The registry names them in `cloud_transports`.
5 changes: 5 additions & 0 deletions contract/registry.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,11 @@ source_states: [live, lagging, stale, down, never]
# tunnel is the box's own session carried by FTW Cloud, which cannot read it.
# cloud is the box-signed copy FTW Cloud holds and can read.
carrier_states: [webrtc, relay, tunnel, cloud, cache, none]
# How a session with FTW Cloud travels, under the tunnel and cloud carriers:
# QUIC on UDP 443, or the WebSocket on TCP 443 that a client falls back to
# (srcfl/ftw docs/protocol-cloud.md, "QUIC"). It says nothing about
# freshness, which stays carrier and source state.
cloud_transports: [quic, websocket]

# ---------------------------------------------------------------------------
# History resolutions backed by what the box actually stores.
Expand Down
150 changes: 150 additions & 0 deletions docs/protocol-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,151 @@ data. The network sees when messages go and how large they are, which shows
which class changed and when. The privacy notice says so (ADR 0008,
decision 5).

## QUIC

The uplink and the app session also run over QUIC (RFC 9000) on UDP 443 at
`cloud.ftw.energy`, with the same certificate as TCP. The messages and every
rule in this document stay the same; only the framing below changes. The
WebSocket stays too, and a client falls back to it when UDP does not get
through. The one-shot read and the extension read are plain HTTPS, which the
service also serves over HTTP/3 (RFC 9114); they need nothing here.
`go/protocol/cloudwire/quic.go` holds the names and the codec, and
`cloud-quic-vectors.json` the bytes.

| ALPN | Connection | WebSocket it stands for |
|---|---|---|
| `ftw-cloud-box/1` | Uplink | `/v1/box` |
| `ftw-cloud-app/1` | App session | `/v1/app?home=<id>` |
| `h3` | HTTP/3 | — |

The service offers only the ALPNs it serves. With the sessions turned off it
offers `h3` alone, so a client's QUIC handshake fails at once and it takes
the WebSocket.

- **TLS 1.3 inside QUIC.** `ekm` is the exporter of that TLS session, with the
label and length of [Handshake](#handshake), and a box binds `hello` to it
as it does over TCP. Nothing may end QUIC in front of the service; a load
balancer that forwards UDP packets is fine.
- **Message.** One WebSocket binary message becomes one message: its length
as a QUIC variable-length integer (RFC 9000 §16), then its bytes. The size
limits of the WebSocket apply. A length over them closes the connection
with 4400.
- **Control stream.** The client opens bidirectional stream 0 and sends the
*target* first, as one message: the path and query the WebSocket would name,
such as `/v1/app?home=<id>`, in ASCII. The service checks it as it checks
the WebSocket's URL. After it, every message of the connection goes on this
stream, both ways, except those below.
- **Bulk stream.** A sender may open one unidirectional stream and send on it,
in order, the `tun` messages whose `d` carries a lane 1 frame of the
phone's session: snapshots, history and answers too large for lane 0. It
opens it the first time it has bulk to send. The box already tells bulk
from lane 0 for its pacing (see [Backpressure](#backpressure-and-limits)).
The service sends a `tun` that came on the box's bulk stream on the phone's
bulk stream. A lost bulk packet then never holds up a command, its answer
or a tick. A phone sends everything on the control stream.
- **Datagrams (RFC 9221).** The box sends `hb`, and `live` without `sources`,
as one QUIC DATAGRAM each when it fits, and otherwise on the control stream.
The service sends those same envelopes to a phone the same way, each as one
Noise transport message. A datagram is never sent again, and the newest
wins. The [freshness](#freshness) rules already cover one that is lost or
late: a phone sees the gap through `hb.liveSeq` and ages what it holds
instead of showing it as live. A `live` that carries `sources` goes on the
control stream, because a lost table would leave sources stale until the
next full `live`. Nothing else goes as a datagram, and tunnel traffic never
does: lane 0 carries commands, their answers and deltas that must arrive,
and two paths for lane 0 would show the service which frames are which.
- The service closes the uplink with 4400 on a datagram that is not `hb`
or a `live` without `sources`. It keeps an `hb` or `live`, on either
path, only when its `seq` is above the newest of its type on this
connection, and drops a late one, with one exception: a late `live`
whose `srcRev` is above the table it holds still gives its table, and
nothing else. A phone does the same (Freshness case 17), so a table
overtaken by a datagram is not lost.
- Each end offers datagrams in its transport parameters. A sender whose
peer does not uses the control stream.
- Until a phone's first transport message arrives, the service sends it
everything on the control stream, behind message 2. quic-go and other
stacks send datagrams ahead of stream data, so an early `hb` could
otherwise reach a phone before the message 2 it needs to read it.
- **Order.** Messages on one stream arrive in order; across streams and
datagrams they may not. Noise transport carries its own counter, one for
all three paths, so a burst on the bulk stream can overtake a control
message by a few hundred. A receiver of a session that can travel over
QUIC keeps a window of at least 1024 (`appwire.ReplayWindow`), so the app
session and the tunnel take messages out of order. It drops a datagram
that falls outside the window or repeats; the same on a stream ends the
session, as over the WebSocket. The service's rule to forward a home's
messages in the order it received them holds per stream.
- **Streams.** A peer that opens a second bidirectional stream, or a second
unidirectional one, breaks the protocol. The service grants one of each.
- **Closing.** A close code becomes the QUIC application error code, with
the WebSocket close reason as the reason phrase. 1000 closes normally.
- **Liveness.** QUIC stands in for the WebSocket ping. The client sends a QUIC
PING after 15 s with nothing to send, which keeps a NAT's UDP mapping
open; many drop one after 30 s. Both ends close a connection silent for
45 s, as the WebSocket's read deadline does.
- **Migration.** A client may move its connection to a new address (RFC 9000
§9): a phone from Wi-Fi to cellular, or a box whose NAT rebinds. The
service validates the new path and follows it. The Noise sessions inside
never notice: no handshake, no new `sid`.

### Racing the WebSocket

A client starts QUIC first. If the QUIC handshake has not finished after
250 ms, it also dials the WebSocket, takes the first that opens and closes
the other. It remembers which won on each network for a day and starts the
next time with that one, giving the other the same 250 ms; after a day QUIC
goes first again, so a network that blocked UDP once is tried again. A QUIC
dial that loses to the WebSocket goes on for up to 3 s, only to learn: if it
opens, QUIC goes first next time, so one lost packet does not keep QUIC
second for a day. The session stays on the WebSocket either way.

A QUIC handshake that finishes proves less than a session that works. A
QUIC connection that ends before the first session message (the `challenge`
for a box, message 2 for a phone), or that the path drops within 60 s of
it, counts as QUIC failing on that network: the client starts there with
the WebSocket for a day, and a learning dial in that day does not move QUIC
first. A close the service sends on purpose, with a code from 1000 up, and a
stateless reset after a restart, do not count. A box names its network by
its local address; a phone by its path's interfaces and gateways, kept as a
hash. A client
says which carries it, by the registry's `cloud_transports`: `quic` or
`websocket`. Core shows it in `GET /api/cloud`; the app shows it beside the
carrier. The carrier stays `tunnel` or `cloud` over both.

### Early data

A client that holds a session ticket from the service may send 0-RTT data. It
may put there only the target and, on the app session, Noise message 1, with
`now` and the box session's message 1 in `now.tun`. Nothing else can go
there: every later message is a Noise transport message, which the client
can encrypt only once message 2 has come back. So no `cmd` or `api.req` ever
rides in early data. A box puts only its target there, and the service's
`challenge` comes back in its first flight.

The network can replay early data. These rules make a replay change nothing:

- **A phone's session waits for the handshake.** The service reads a phone's
message 1 from early data but acts on it only once the QUIC handshake has
finished, which the sender of a replay cannot do. Until then nothing is
forwarded, opened or answered: no session is replaced, no pairing code
spent, no `watch` sent. This holds for the first copy too, which matters
when the network dropped the original and the phone dialled again.
- **A box's target is all it sends early.** The service may answer it with a
`challenge` before the handshake ends. A challenge changes nothing, and
only the box can answer it with a `hello`.
- **For a day.** A ticket allows early data for 24 hours after the service
issued it, and resumption without early data after that. A restart draws
new ticket keys, so every client makes a full handshake once.
- **The box holds a tunnel session as pending** until its first transport
message decrypts (see [Tunnel](#tunnel)), so a `now.tun` takes no slot
and spends no code, whoever sent it.
- **HTTP/3 requests wait for the handshake**, since no plain route has said
which of its requests survive a replay.

A client whose early data the service rejects sends it again once the
handshake is done.

## Envelope and signature

```
Expand Down Expand Up @@ -1165,6 +1310,11 @@ In all of them the phone uses rule N with 100 ms in transit unless stated.
the generic line with no buttons.
16. **The box's state wins.** The box reports a source as `down`, and t = 1 s.
The source shows `down`.
17. **A late table still counts.** The box changes the table: a `live` with
`seq` 2 and `srcRev` 2 goes on the stream, and a `live` without a table,
`seq` 3 and `srcRev` 2, overtakes it as a datagram. The source shows
`stale` (rule 1) until the late `live` arrives; the phone then takes its
table, keeps `seq` 3's values, and the source shows `live`.

## Grant on the box

Expand Down
3 changes: 1 addition & 2 deletions go/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ require (
github.com/gorilla/websocket v1.5.3
github.com/mochi-mqtt/server/v2 v2.7.9
github.com/parquet-go/parquet-go v0.32.0
github.com/quic-go/quic-go v0.63.0
github.com/shirou/gopsutil/v4 v4.26.8
github.com/simonvetter/modbus v1.6.4
github.com/yuin/gopher-lua v1.1.2
Expand All @@ -21,12 +22,10 @@ require (
)

require (
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/go-playground/locales v0.12.1 // indirect
github.com/go-playground/universal-translator v0.16.0 // indirect
github.com/gorilla/mux v1.8.1 // indirect
github.com/leodido/go-urn v1.1.0 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/relvacode/iso8601 v1.6.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
gopkg.in/go-playground/validator.v9 v9.30.0 // indirect
Expand Down
16 changes: 10 additions & 6 deletions go/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ github.com/caarlos0/env/v11 v11.3.1 h1:cArPWC15hWmEt+gWk7YBi7lEXTXCvpaSdCiZE2X5m
github.com/caarlos0/env/v11 v11.3.1/go.mod h1:qupehSf/Y0TUTsxKywqRt/vJjN5nz6vauiYEUUr8P4U=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/ebitengine/purego v0.10.2 h1:W809HbnvzAxgdm+aOvlSekrM16wGCdT/e76+9tS7gzE=
Expand Down Expand Up @@ -78,10 +76,12 @@ github.com/parquet-go/parquet-go v0.32.0/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLva
github.com/pierrec/lz4/v4 v4.1.25 h1:kocOqRffaIbU5djlIBr7Wh+cx82C0vtFb0fOurZHqD0=
github.com/pierrec/lz4/v4 v4.1.25/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/quic-go/go-ossfuzz-seeds v0.1.0 h1:APacT+iIaNF6fd8AGEiN3bT/Jtkd2jz4v4TzM7MFjy0=
github.com/quic-go/go-ossfuzz-seeds v0.1.0/go.mod h1:3IOHRbJIc+L6YKMwfDtJAM9Vj9k0YY4muhuyUYk5tbk=
github.com/quic-go/quic-go v0.63.0 h1:LIFGHI4PFUhhw2dDD1ARHdCff143ffMHwZtbnbuJ78A=
github.com/quic-go/quic-go v0.63.0/go.mod h1:RAro2j2yN9a9EiPACLHT9IB2NXCvGQmmo/alT0yYI0w=
github.com/relvacode/iso8601 v1.6.0 h1:eFXUhMJN3Gz8Rcq82f9DTMW0svjtAVuIEULglM7QHTU=
github.com/relvacode/iso8601 v1.6.0/go.mod h1:FlNp+jz+TXpyRqgmM7tnzHHzBnz776kmAH2h3sZCn0I=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
Expand All @@ -103,8 +103,8 @@ github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/tklauser/go-sysconf v0.3.16 h1:frioLaCQSsF5Cy1jgRBrzr6t502KIIwQ0MArYICU0nA=
github.com/tklauser/go-sysconf v0.3.16/go.mod h1:/qNL9xxDhc7tx3HSRsLWNnuzbVfh3e7gh/BmM179nYI=
github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9RXw=
Expand All @@ -119,6 +119,10 @@ github.com/yuin/gopher-lua v1.1.2 h1:yF/FjE3hD65tBbt0VXLE13HWS9h34fdzJmrWRXwobGA
github.com/yuin/gopher-lua v1.1.2/go.mod h1:7aRmXIWl37SqRf0koeyylBEzJ+aPt8A+mmkQ4f1ntR8=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/mod v0.38.0 h1:MECBjubtXD7yj4HrhIUcywNaGeNVUdfVnxmPajOk4yk=
Expand Down
16 changes: 16 additions & 0 deletions go/internal/appproto/gencontract/gencontract.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ type Registry struct {
// with FTW Cloud, and what for.
CloudScopes []string `yaml:"cloud_scopes"`
CloudPurposes []string `yaml:"cloud_purposes"`
// CloudTransports are what a session with FTW Cloud travels over.
CloudTransports []string `yaml:"cloud_transports"`
// AgentArgs is what an agent may put in each op it can name, and
// Reads the reads it may send with the scope each needs.
AgentArgs map[string][]struct {
Expand Down Expand Up @@ -338,6 +340,20 @@ func writeCloud(b *bytes.Buffer, reg Registry) {
fmt.Fprintf(b, "\tCloudPurpose%s,\n", goName(p))
}
b.WriteString("}\n\n")

b.WriteString("// What a session with FTW Cloud travels over, under its carrier.\n")
b.WriteString("const (\n")
for _, t := range reg.CloudTransports {
fmt.Fprintf(b, "\tCloudTransport%s = %q\n", goName(t), t)
}
b.WriteString(")\n\n")

b.WriteString("// CloudTransports lists them in the registry's order.\n")
b.WriteString("var CloudTransports = []string{\n")
for _, t := range reg.CloudTransports {
fmt.Fprintf(b, "\tCloudTransport%s,\n", goName(t))
}
b.WriteString("}\n\n")
}

// writeOps emits the operation table as data rather than constants. The op
Expand Down
Loading
Loading