From b2ac73e4024ee6e9ebbe7fbb960691e00f4dcd34 Mon Sep 17 00:00:00 2001 From: Fredrik Ahlgren Date: Thu, 15 Oct 2026 16:47:14 +0200 Subject: [PATCH 1/4] feat: carry each phone's session to its box, and survive a restart Step 1 of the one-door plan, the ftw-cloud half; pairs with srcfl/ftw feat/cloud-tunnel, whose go/protocol this pins. - tun: a phone's tun {sid, d} on /v1/app goes to its box as tun {dev, sid, d}, with dev from the roster entry the service's own handshake matched; the box's answers go back to that device's session. Routed per device on queues of their own, never as a held class. Only a box that lists tun in hello.caps gets any. An empty d ends a session at either end, and the service sends one when a phone's app session or the box's uplink ends. now.tun lets the box session ride in message 1. - Admission: a roster device needs tun or a readable copy. A pending pairing hash admits one new phone, which may only tunnel and takes its roster entry when the box pairs it. - Limits: per home and direction 240 tun then 60/s and 4 MiB then 512 KiB/s, dropped and counted past that; frames up to MaxTunFrame; Origin https://app.ftw.energy allowed on /v1/app. - The copy: readings are held only while the grant serves; a withdrawn grant deletes the copy and the tokens and gets purged {what: copy}. - Before commands move here: rosters are read back from the registry at start; box.unreachable waits survive a restart; a connection becomes the home's only after hello_ok, so watch or tun never precede it. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Fredrik Ahlgren --- CLAUDE.md | 37 +- README.md | 56 ++- cmd/ftwcloudd/main.go | 12 + go.mod | 2 +- go.sum | 4 +- internal/appsession/appsession.go | 158 ++++++- internal/appsession/fuzz_test.go | 29 +- internal/push/push.go | 79 +++- internal/push/resume_test.go | 74 +++ internal/registry/push.go | 18 + internal/registry/registry.go | 74 +++ internal/server/e2e_test.go | 6 +- internal/server/tunnel_e2e_test.go | 700 +++++++++++++++++++++++++++++ internal/store/session.go | 101 ++++- internal/store/store.go | 136 +++++- internal/store/store_test.go | 3 +- internal/uplink/downlink.go | 16 +- internal/uplink/tunnel_test.go | 205 +++++++++ internal/uplink/uplink.go | 217 ++++++++- internal/uplink/uplink_test.go | 29 +- 20 files changed, 1856 insertions(+), 100 deletions(-) create mode 100644 internal/push/resume_test.go create mode 100644 internal/server/tunnel_e2e_test.go create mode 100644 internal/uplink/tunnel_test.go diff --git a/CLAUDE.md b/CLAUDE.md index 04e4921..e784d9f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,7 +1,8 @@ # FTW Cloud guide -`ftwcloudd` keeps a signed, readable copy of each home that turned FTW Cloud -on, and serves it to that home's phones. Read [README.md](README.md) first, +`ftwcloudd` is the one door between a home's phones and its box: it carries +each phone's own Noise session with the box, and keeps a signed, readable +copy of each home whose owner turned that on. Read [README.md](README.md) first, then the normative [FTW Cloud protocol](https://github.com/srcfl/ftw/blob/master/docs/protocol-cloud.md) and [ADR 0008](https://github.com/srcfl/ftw/blob/master/docs/adr/0008-ftw-cloud.md) @@ -10,22 +11,29 @@ maintains it. ## Invariants -- **FTW Cloud reads and never controls.** It sends a box only `challenge`, - `hello_ok`, `watch`, `resync` and `purged` (`internal/uplink/downlink.go`). - Commands go from the app to the box over Noise, never through here. +- **FTW Cloud forwards ciphertext it cannot read, and never chooses an + action.** It sends a box only `challenge`, `hello_ok`, `watch`, `resync`, + `purged` and `tun` (`internal/uplink/downlink.go`). Commands go from the app + to the box inside the phone's own Noise session, which rides here as `tun` + under the device id the service's own handshake proved, never one a phone + names. - **One way to a home:** `store.For(principal)`. No listing, no second lookup, no principal for staff. A new route needs a row in `server.Routes` and must pass `TestNoRouteReadsAHome`. - **The home id comes from the key that signed hello** on this connection, never from a body field. Every later box message must verify with that key. -- **No readings on disk.** Memory holds readings; SQLite holds only the home - id, the box key, the signed roster, owners' suspensions, tombstones, APNs - registrations and the hashes of extension tokens. Nothing logs addresses. -- **A phone can only narrow.** On `/v1/app` it can say whether it is looking, - and an owner can suspend a roster device. On `/v1/app/once` a watch can - read, ask for its token and register for push, and nothing else. The - service admits a phone or a watch by the key its Noise handshake proved, - and only if the box's roster lists it. +- **No readings on disk.** Memory holds readings, and only while the roster's + grant serves a copy; SQLite holds only the home id, the box key, the signed + roster, owners' suspensions, tombstones, APNs registrations, the hashes of + extension tokens and when each away box's alert is due. Nothing logs + addresses. +- **A phone can only narrow, or talk to its own box.** On `/v1/app` it can + say whether it is looking, an owner can suspend a roster device, and a + device the roster lists with `tun` can send its box session as `tun`. On + `/v1/app/once` a watch can read, ask for its token and register for push, + and nothing else. The service admits a phone or a watch by the key its + Noise handshake proved, and only if the box's roster lists it, or once for + a pairing code the roster lists by its hash. - **TLS ends in `ftwcloudd`.** The handshake is bound to the TLS session; a terminating proxy breaks it on purpose. A TCP load balancer is fine. - **Forward the box's bytes unchanged.** The service cannot sign for a box, @@ -39,7 +47,8 @@ maintains it. - A wire change is a pair: `srcfl/ftw` (`go/protocol` and its golden vectors) first, then this repository, moving the pin in `go.mod`. Say so in both PRs. - Keep the service small. The budget is about 3,000 lines of Go; tiles go - first if it is overrun. + first if it is overrun. It is over today; the one-door plan gives each part + a budget of its own. - Nothing is deployed from here without the owner. `deploy/` describes the running host; it does not run itself. - Keep task breakdowns and agent notes out of the repository. diff --git a/README.md b/README.md index b72dad5..b4b4715 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,10 @@ # FTW Cloud -FTW Cloud keeps a signed, readable copy of each home that turns it on, and -serves it to that home's phones. It reads and never controls. The box opens -the connection, picks and signs every message, and can revoke at any time. +FTW Cloud is the one door between a home's phones and its box. It carries +each phone's own Noise session with the box as ciphertext it cannot read or +forge, and keeps a signed, readable copy of each home whose owner turns that +on. It never chooses an action. The box opens the connection, picks and signs +every message but the phones' ciphertext, and can revoke at any time. This is `ftwcloudd`, the service. The decision is [ADR 0008](https://github.com/srcfl/ftw/blob/master/docs/adr/0008-ftw-cloud.md) @@ -22,9 +24,33 @@ to it yet. the box sent. Readings are dropped an hour after the box goes quiet and are never written to disk. - The registry (SQLite) holds the home id, the box key, the signed roster, - owners' suspensions, APNs registrations and the hashes of extension - tokens. A box that turns FTW Cloud off gets its home deleted - and a signed receipt. + owners' suspensions, APNs registrations, the hashes of extension tokens + and when an away box's alert is due. A box whose owner turns the readable + copy off gets that copy deleted and a signed receipt (`purged {what: + "copy"}`); a box whose link ends gets its home deleted and a receipt. +- **A restart forgets no one.** At start the service reads every signed + roster back from the registry and checks it with the home's key, so a + home's phones are admitted before its box reconnects, and the wait for + `box.unreachable` resumes where it was. A connection becomes the home's + only after `hello_ok` has gone, so nothing reaches a box ahead of it. +- **The tunnel.** A box that lists `tun` in `hello.caps` takes its phones' + sessions: a phone sends `tun {sid, d}` on its app session, where `d` is its + `ftw.session.v1` session with the box byte for byte; the service adds the + device id of the roster entry its own handshake matched and sends `tun + {dev, sid, d}` down `/v1/box`, and the box's answers come back the same + way. The box session's first message may ride in the app session's message + 1 (`now.tun`), so it is up one round trip after TLS. An empty `d` ends a + session: the box hears when the phone's app session ends, and the phone + when the box's uplink does. tun has its own queues and limits per home and + direction: 240 messages then 60 a second, 4 MiB then 512 KiB a second. A + roster device needs `tun` or a readable copy to be admitted at all. +- **Pairing through the service.** While a QR code is on the box's screen, + its roster lists the code's hash. A new phone proves the hash in `now.pair` + and is admitted once, as a device that may only open a tunnel; the box + checks the code itself inside the phone's handshake. When the roster lists + the new phone its session takes that entry and stays open. +- The web app may open `/v1/app` from `https://app.ftw.energy`; any other + `Origin` is refused. - `/source` serves the source of the running build, as AGPL-3.0 asks, and names its commit and image digest. - A phone reads its home at `/v1/app?home=`: Noise IK to the service's @@ -95,7 +121,7 @@ Not built yet: box ──wss, TLS 1.3, signed frames──▶ /v1/box ──▶ store (memory) ◀── store.For(principal) ◀── /v1/app ◀──Noise IK── phone │ ◀── /v1/app/once ◀──Noise IK in one POST── watch │ ◀── /v1/now ◀──token──── widget - └────▶ registry (SQLite: key, roster, tombstones, suspensions, APNs, token hashes) + └────▶ registry (SQLite: key, roster, tombstones, suspensions, APNs, token hashes, away) ``` - **The home id is a hash of the box's key.** It comes from the key that @@ -107,13 +133,15 @@ box ──wss, TLS 1.3, signed frames──▶ /v1/box ──▶ store (memory) principal reads only `live`, `prices` and `hb` within that same scope. There is no listing and no principal for staff; `TestNoRouteReadsAHome` checks every route. -- **A phone or a watch can narrow and nothing else.** A phone can say - whether it is looking, and an owner can suspend a device in the roster. A - watch can read, ask for its token and register for push. Commands go to - the box over the relay, never through here. -- **The service sends five things.** `challenge`, `hello_ok`, `watch`, - `resync` and `purged` are the whole downlink, and none asks the box to act. - The box closes the uplink on anything else. +- **A phone or a watch can narrow, and talk to its own box.** A phone can + say whether it is looking, an owner can suspend a device in the roster, and + a phone sends its box session as `tun`, which the service cannot read. A + watch can read, ask for its token and register for push. +- **The service sends six things.** `challenge`, `hello_ok`, `watch`, + `resync`, `purged` and `tun` are the whole downlink. The first five ask + the box for nothing; `tun` carries a phone's ciphertext, which the box + judges as it judges the same session over the relay. The box closes the + uplink on anything else. - **No TLS terminator in front.** Box and service bind the handshake to the TLS session, so `ftwcloudd` ends TLS itself. A TCP load balancer is fine. - **Root on the host can read memory.** The privacy notice says so. diff --git a/cmd/ftwcloudd/main.go b/cmd/ftwcloudd/main.go index 2b47263..2da6f0f 100644 --- a/cmd/ftwcloudd/main.go +++ b/cmd/ftwcloudd/main.go @@ -99,6 +99,14 @@ func serve(args []string) error { st := store.New(nil) st.UseSuspensions(reg.Suspended) up := uplink.New(st, reg, key, log) + // A restart is a deploy, and commands ride through here: phones are + // admitted by the rosters their boxes signed before, without waiting + // for every box to reconnect. + if n, err := up.Restore(); err != nil { + return err + } else if n > 0 { + log.Info("rosters restored from the registry", "homes", n) + } tokens := token.New(st, reg, log) srv := &server.Server{Store: st, Registry: reg, Uplink: up, Tokens: tokens, Log: log} @@ -134,6 +142,7 @@ func serve(args []string) error { srv.App = appsession.New(st, reg, up, noise, log) srv.App.Push = pushSvc srv.App.Tokens = tokens + srv.App.Tunnel = up } app := srv.App up.OnConnect = func(home string) { @@ -179,6 +188,9 @@ func serve(args []string) error { defer stop() go srv.Upkeep(ctx, time.Minute) go pushSvc.Run(ctx) + if err := pushSvc.Resume(); err != nil { + log.Warn("could not restart the dead man's switch", "err", err) + } if *devPush != "" { if err := serveDevPush(ctx, *devPush, reg, pushSvc, sender, log); err != nil { return err diff --git a/go.mod b/go.mod index e48952a..48c660f 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.26.0 require ( github.com/fxamacker/cbor/v2 v2.9.4 github.com/gorilla/websocket v1.5.3 - github.com/srcfl/ftw/go/protocol v0.0.0-20261014110822-7c72a5b33aa0 + github.com/srcfl/ftw/go/protocol v0.0.0-20261015144614-0a5e04691065 golang.org/x/crypto v0.57.0 golang.org/x/sys v0.48.0 modernc.org/sqlite v1.59.0 diff --git a/go.sum b/go.sum index be93d77..17c3405 100644 --- a/go.sum +++ b/go.sum @@ -16,8 +16,8 @@ github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOF github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= -github.com/srcfl/ftw/go/protocol v0.0.0-20261014110822-7c72a5b33aa0 h1:4p9TFWdkIk5y2fK3/9FOeUV9AukTjIBd7jQZDLG8Qx8= -github.com/srcfl/ftw/go/protocol v0.0.0-20261014110822-7c72a5b33aa0/go.mod h1:VEjr/o0yZijJ1nApuJK+nak6FxJ/fubAUBNOunqNrUU= +github.com/srcfl/ftw/go/protocol v0.0.0-20261015144614-0a5e04691065 h1:tKaqxGV1K69ljIBH1szkDz3zxqeZKdu5eanSeDY9qNo= +github.com/srcfl/ftw/go/protocol v0.0.0-20261015144614-0a5e04691065/go.mod h1:VEjr/o0yZijJ1nApuJK+nak6FxJ/fubAUBNOunqNrUU= github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M= diff --git a/internal/appsession/appsession.go b/internal/appsession/appsession.go index 0e7b84c..e0e8d08 100644 --- a/internal/appsession/appsession.go +++ b/internal/appsession/appsession.go @@ -10,8 +10,10 @@ // the device may read goes out unchanged, one frame per Noise message. // // A phone may ask for faster updates, ask for its extension token, register -// for alerts, and an owner may suspend a device. That is all it can do here: -// commands go to the box over the relay, never through FTW Cloud. +// for alerts, and an owner may suspend a device. Its own Noise session with +// the box rides here as tun, which the service forwards under the device id +// its handshake proved and can neither read nor forge. That is all it can +// do here. package appsession import ( @@ -31,9 +33,12 @@ import ( ) const ( - // maxPhoneMessage bounds one message from a phone. Message 1 with now is - // about 120 bytes and nothing a phone sends later is larger. - maxPhoneMessage = 4 << 10 + // maxPhoneMessage bounds one message from a phone: a tun around the + // largest bulk frame. Every other message must fit maxPlainMessage. + maxPhoneMessage = cloudwire.MaxTunFrame + appwire.TransportOverhead + // maxPlainMessage bounds every phone message but tun. Message 1 with + // now is about 160 bytes. + maxPlainMessage = 4 << 10 // handshakeTimeout bounds message 1. handshakeTimeout = 10 * time.Second // The same ping, read and write deadlines as the uplink. @@ -44,6 +49,11 @@ const ( // suspends a device. Past this burst at this rate the session closes. rateBurst = 20 ratePerSec = 2 + // tun counts apart, against the home's limits at the uplink, which drop + // what is over. A phone that sends twice as fast as those pass is not a + // phone at work, and its session closes. + tunBurst = 480 + tunPerSec = 120 // maxNoiseMessage is the largest Noise message, handshake or transport. maxNoiseMessage = 0xffff // WatchRenew is how often the service asks the box again for one-second @@ -62,6 +72,16 @@ type Watcher interface { Watch(home string) } +// Tunneler sends a phone's tun to its home's box. The uplink is one. False +// means no box takes it now. +type Tunneler interface { + Tun(home string, t cloudwire.Tun) bool +} + +// WebOrigin is the one browser origin the service accepts on /v1/app: the +// web app's. The phone and the box send no Origin. +const WebOrigin = "https://app.ftw.energy" + // Server is the /v1/app handler. type Server struct { store *store.Store @@ -85,6 +105,9 @@ type Server struct { // Tokens issues extension tokens. Nil ignores token.get. Set before // serving. Tokens Tokens + // Tunnel carries the phones' sessions to their boxes. Nil answers every + // tun with its end. Set before serving. + Tunnel Tunneler mu sync.Mutex looking map[string]*look @@ -127,9 +150,12 @@ func New(st *store.Store, reg *registry.Registry, w Watcher, key appwire.KeyPair var upgrader = websocket.Upgrader{ EnableCompression: false, - // The app sends no Origin. A browser always does, and has no business - // here. - CheckOrigin: func(r *http.Request) bool { return r.Header.Get("Origin") == "" }, + // The app sends no Origin. A browser always does, and only the web app's + // has business here. + CheckOrigin: func(r *http.Request) bool { + o := r.Header.Get("Origin") + return o == "" || o == WebOrigin + }, } func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { @@ -166,7 +192,7 @@ func (s *Server) serve(conn *websocket.Conn, home string) (int, error) { if err != nil { return 0, err } - if kind != websocket.BinaryMessage { + if kind != websocket.BinaryMessage || len(msg1) > maxPlainMessage { return cloudwire.CloseProtocol, errProtocol } hs, err := appwire.NewResponder(appwire.HandshakeOptions{StaticKey: s.key, Prologue: cloudwire.AppPrologue(home)}) @@ -179,12 +205,13 @@ func (s *Server) serve(conn *websocket.Conn, home string) (int, error) { // query. Neither says anything about the home. return cloudwire.CloseDenied, err } - hz, err := parseNow(payload) + now, err := parseNow(payload) if err != nil { return cloudwire.CloseProtocol, err } + hz := now.Hz - dev, err := s.store.For(store.Principal{Kind: store.Device, Home: home, DeviceKey: hs.RemoteStatic()}) + dev, err := s.store.For(store.Principal{Kind: store.Device, Home: home, DeviceKey: hs.RemoteStatic(), Pair: now.Pair}) if err != nil { return cloudwire.CloseDenied, err } @@ -206,7 +233,14 @@ func (s *Server) serve(conn *websocket.Conn, home string) (int, error) { return websocket.CloseInternalServerErr, err } c := &phoneConn{s: s, conn: conn, dev: dev, home: home, tokens: rateBurst, refilled: s.now(), + tunTokens: tunBurst, tunAt: s.now(), sess: sess, replies: make(chan []byte, 1), quit: make(chan struct{})} + // The box session's message 1, when it rode in message 1, goes to the + // box before message 2 goes back: the box's answer then follows message + // 2 by one leg, and the session is up one round trip after TLS. + if now.Tun != nil { + c.tunEarly(*now.Tun) + } if err := c.write(msg2); err != nil { return 0, err } @@ -223,20 +257,21 @@ func (s *Server) serve(conn *websocket.Conn, home string) (int, error) { c.setLooking(hz >= cloudwire.HzLooking) defer c.stopLooking() + defer c.endTun() return c.run(sess) } // parseNow reads message 1's payload. -func parseNow(payload []byte) (int, error) { +func parseNow(payload []byte) (cloudwire.Now, error) { env, err := cloudwire.DecodeEnvelope(payload) if err != nil || env.T != cloudwire.MsgNow { - return 0, errProtocol + return cloudwire.Now{}, errProtocol } var now cloudwire.Now if err := cloudwire.Unmarshal(env.B, &now); err != nil { - return 0, errProtocol + return cloudwire.Now{}, errProtocol } - return now.Hz, nil + return now, nil } // snapshotPayload is message 2. Envelopes that do not fit in a message 2 of @@ -338,12 +373,23 @@ type phoneConn struct { tokens float64 refilled time.Time + // tunTokens is the tun bucket, refilled at tunAt. + tunTokens float64 + tunAt time.Time // replies carries the service's answers from the read loop to the // session's goroutine, the only one that writes; quit closes when that // goroutine is done. replies chan []byte quit chan struct{} + + // sess is the store's session; tunSid the box session this phone ran + // last through the tunnel, when hasTun. Touched by the read loop and, + // once it is done, by the session's end. + sess *store.Session + tunMu sync.Mutex + tunSid uint32 + hasTun bool } func (c *phoneConn) setLooking(on bool) { @@ -418,6 +464,14 @@ func (c *phoneConn) run(sess *store.Session) (int, error) { if err := c.forward(env); err != nil { return 0, err } + case t := <-sess.Tun(): + env, err := cloudwire.Unsigned(cloudwire.MsgTun, t) + if err != nil { + return 0, err + } + if err := c.forward(env); err != nil { + return 0, err + } case <-sess.Done(): return sess.Code(), nil case r := <-read: @@ -448,9 +502,6 @@ func (c *phoneConn) readLoop() (int, error) { return 0, err } _ = c.conn.SetReadDeadline(time.Now().Add(readTimeout)) - if !c.allow() { - return cloudwire.CloseProtocol, errProtocol - } if kind != websocket.BinaryMessage { return cloudwire.CloseProtocol, errProtocol } @@ -464,6 +515,59 @@ func (c *phoneConn) readLoop() (int, error) { } } +// tun forwards the phone's own session to its box, under the device id the +// handshake proved. A device without tun in the roster sends nothing. When +// no box takes it, the phone hears its session ended. +func (c *phoneConn) tun(t cloudwire.AppTun) error { + dev := c.dev.Device() + if !dev.Tun || len(t.D) > cloudwire.TunDataMax { + return nil + } + c.tunMu.Lock() + c.tunSid, c.hasTun = t.Sid, len(t.D) > 0 + c.tunMu.Unlock() + if c.sess != nil && len(t.D) > 0 { + c.sess.SetSid(t.Sid) + } + if c.s.Tunnel != nil && c.s.Tunnel.Tun(c.home, cloudwire.Tun{Dev: dev.ID, Sid: t.Sid, D: t.D}) { + return nil + } + if len(t.D) == 0 { + return nil + } + return c.say(cloudwire.MsgTun, cloudwire.AppTun{Sid: t.Sid}) +} + +// tunEarly forwards the box session's message 1 that rode in message 1. No +// reply can go yet, so a box that cannot take it is heard about as the end +// of the session, once the stream runs. +func (c *phoneConn) tunEarly(t cloudwire.AppTun) { + dev := c.dev.Device() + if !dev.Tun || len(t.D) == 0 || len(t.D) > cloudwire.TunDataMax { + return + } + c.tunMu.Lock() + c.tunSid, c.hasTun = t.Sid, true + c.tunMu.Unlock() + c.sess.SetSid(t.Sid) + if c.s.Tunnel != nil && c.s.Tunnel.Tun(c.home, cloudwire.Tun{Dev: dev.ID, Sid: t.Sid, D: t.D}) { + return + } + c.sess.Ended(t.Sid) +} + +// endTun tells the box that the phone's session through here is over, when +// the app session ends. +func (c *phoneConn) endTun() { + c.tunMu.Lock() + sid, had := c.tunSid, c.hasTun + c.hasTun = false + c.tunMu.Unlock() + if had && c.s.Tunnel != nil { + c.s.Tunnel.Tun(c.home, cloudwire.Tun{Dev: c.dev.Device().ID, Sid: sid}) + } +} + // apply acts on one message from the phone. Types it does not know are // ignored, so a newer app works with an older service. func (c *phoneConn) apply(frame []byte) (int, error) { @@ -475,6 +579,24 @@ func (c *phoneConn) apply(frame []byte) (int, error) { if err != nil { return cloudwire.CloseProtocol, err } + if env.T == cloudwire.MsgTun { + now := c.s.now() + c.tunTokens = min(tunBurst, c.tunTokens+now.Sub(c.tunAt).Seconds()*tunPerSec) + c.tunAt = now + if c.tunTokens < 1 { + return cloudwire.CloseProtocol, errProtocol + } + c.tunTokens-- + var t cloudwire.AppTun + if err := cloudwire.Unmarshal(env.B, &t); err != nil { + return cloudwire.CloseProtocol, err + } + return 0, c.tun(t) + } + // tun has limits of its own, per home; everything else counts here. + if len(frame) > maxPlainMessage || !c.allow() { + return cloudwire.CloseProtocol, errProtocol + } switch env.T { case cloudwire.MsgWatch: var w cloudwire.Watch diff --git a/internal/appsession/fuzz_test.go b/internal/appsession/fuzz_test.go index dd368e7..5ce1be7 100644 --- a/internal/appsession/fuzz_test.go +++ b/internal/appsession/fuzz_test.go @@ -8,13 +8,28 @@ import ( "github.com/srcfl/ftw/go/protocol/cloudwire" ) +// tunRecorder is the uplink, as the fuzz sees it: what reached the box. +type tunRecorder struct{ got []cloudwire.Tun } + +func (r *tunRecorder) Tun(_ string, t cloudwire.Tun) bool { + r.got = append(r.got, t) + return true +} + // Whatever a phone sends after the handshake, the service neither panics nor // does more than the protocol allows: a phone can change whether it looks, -// and an owner can suspend a device in the roster. Nothing else changes. +// an owner can suspend a device in the roster, and a tun reaches the box +// under the device the handshake proved, never one the phone named. Nothing +// else changes. func FuzzPhoneMessage(f *testing.F) { h := newHarness(f) h.live(true) + rec := &tunRecorder{} + h.app.Tunnel = rec for _, seed := range [][]byte{ + mustFrame(cloudwire.MsgTun, cloudwire.AppTun{Sid: 1, D: []byte{1, 2, 3}}), + mustFrame(cloudwire.MsgTun, map[string]any{"dev": "owner", "sid": 2, "d": []byte{4}}), + mustFrame(cloudwire.MsgTun, cloudwire.AppTun{Sid: 3}), mustFrame(cloudwire.MsgWatch, cloudwire.Watch{Hz: 1}), mustFrame(cloudwire.MsgWatch, cloudwire.Watch{Hz: 0}), mustFrame(cloudwire.MsgSuspend, cloudwire.Suspend{DeviceID: "viewer"}), @@ -32,11 +47,18 @@ func FuzzPhoneMessage(f *testing.F) { if err != nil { t.Fatal(err) } - c := &phoneConn{s: h.app, dev: dev, home: h.home, tokens: rateBurst, refilled: h.app.now()} + c := &phoneConn{s: h.app, dev: dev, home: h.home, tokens: rateBurst, refilled: h.app.now(), + tunTokens: tunBurst, tunAt: h.app.now()} + rec.got = nil code, err := c.apply(frame) if err != nil && code != cloudwire.CloseProtocol && code != 0 { t.Fatalf("closed with %d", code) } + for _, sent := range rec.got { + if sent.Dev != dev.Device().ID || !dev.Device().Tun { + t.Fatalf("a tun reached the box as %q from %q", sent.Dev, dev.Device().ID) + } + } c.stopLooking() // Only the owner may suspend, and only the viewer exists to be // suspended besides the owner itself. @@ -59,6 +81,9 @@ func FuzzPhoneMessage(f *testing.F) { func FuzzNow(f *testing.F) { now, _ := cloudwire.Unsigned(cloudwire.MsgNow, cloudwire.Now{Hz: 1}) f.Add(now) + pair, _ := cloudwire.Unsigned(cloudwire.MsgNow, cloudwire.Now{Hz: 1, Pair: make([]byte, 32), + Tun: &cloudwire.AppTun{Sid: 1, D: []byte{1}}}) + f.Add(pair) f.Add([]byte{0xa0}) f.Fuzz(func(t *testing.T, payload []byte) { _, _ = parseNow(payload) diff --git a/internal/push/push.go b/internal/push/push.go index ca6f433..7a643b0 100644 --- a/internal/push/push.go +++ b/internal/push/push.go @@ -317,7 +317,6 @@ func pricedUntil(p cloudwire.Prices) int64 { // BoxConnected stops the dead man's switch for a home whose box is back. func (s *Service) BoxConnected(home string) { s.mu.Lock() - defer s.mu.Unlock() st := s.state(home) st.gen++ st.online, st.told = true, false @@ -325,6 +324,10 @@ func (s *Service) BoxConnected(home string) { st.stop() st.stop = nil } + s.mu.Unlock() + if err := s.reg.AwayClear(home); err != nil { + s.log.Warn("could not record a box's return", "err", err) + } } // BoxGone starts the dead man's switch. why is the box's bye reason, empty @@ -332,30 +335,85 @@ func (s *Service) BoxConnected(home string) { // back until the box has been away for cloudwire.UpdateGraceMs; a revoke // ends it, since the home is gone. func (s *Service) BoxGone(home, why string) { - s.mu.Lock() - defer s.mu.Unlock() if why == cloudwire.ByeRevoked { + s.mu.Lock() if st := s.homes[home]; st != nil && st.stop != nil { st.stop() } delete(s.homes, home) + s.mu.Unlock() + if err := s.reg.AwayClear(home); err != nil { + s.log.Warn("could not forget a revoked home's absence", "err", err) + } return } - st := s.state(home) - st.gen++ - st.online = false - if st.stop != nil { - st.stop() - } wait := time.Duration(cloudwire.UnreachableAfterMs) * time.Millisecond if why == cloudwire.ByeUpdate { wait = time.Duration(cloudwire.UpdateGraceMs) * time.Millisecond } s.log.Info("a box went away", "bye", why, "out_of_reach_after", wait) + // Kept on disk, so a restart in the meantime neither forgets the alert + // nor sends it twice. + if err := s.reg.AwaySet(home, s.now().Add(wait)); err != nil { + s.log.Warn("could not record a box's absence", "err", err) + } + s.arm(home, wait) +} + +// arm starts the wait for box.unreachable. The home's generation moves, so +// a timer from an earlier absence cannot fire into this one. +func (s *Service) arm(home string, wait time.Duration) { + s.mu.Lock() + defer s.mu.Unlock() + st := s.state(home) + st.gen++ + st.online, st.told = false, false + if st.stop != nil { + st.stop() + } gen := st.gen st.stop = s.after(wait, func() { s.unreachable(home, gen) }) } +// Resume starts the dead man's switch again after a restart. A home recorded +// as away waits only what was left of its wait, and one already told is not +// told again. A home with alert registrations and no record was connected +// when the service stopped; its box gets the whole wait to come back. +func (s *Service) Resume() error { + away, err := s.reg.AwayAll() + if err != nil { + return err + } + seen := map[string]bool{} + now := s.now() + for _, a := range away { + seen[a.Home] = true + if a.Told { + s.mu.Lock() + st := s.state(a.Home) + st.online, st.told = false, true + s.mu.Unlock() + continue + } + s.arm(a.Home, max(0, a.Due.Sub(now))) + } + homes, err := s.reg.PushHomes(cloudwire.PushKindAlert) + if err != nil { + return err + } + wait := time.Duration(cloudwire.UnreachableAfterMs) * time.Millisecond + for _, home := range homes { + if seen[home] { + continue + } + if err := s.reg.AwaySet(home, now.Add(wait)); err != nil { + return err + } + s.arm(home, wait) + } + return nil +} + func (s *Service) unreachable(home string, gen uint64) { s.mu.Lock() st := s.homes[home] @@ -365,6 +423,9 @@ func (s *Service) unreachable(home string, gen uint64) { } st.told, st.stop = true, nil s.mu.Unlock() + if err := s.reg.AwayTold(home); err != nil { + s.log.Warn("could not record box.unreachable as sent", "err", err) + } s.log.Info("a box is out of reach; telling its phones") payload, err := cloudwire.AlertPayload(cloudwire.Alert{Home: home, Kind: protocol.PushBoxUnreachable}) if err != nil { diff --git a/internal/push/resume_test.go b/internal/push/resume_test.go new file mode 100644 index 0000000..a34dfbc --- /dev/null +++ b/internal/push/resume_test.go @@ -0,0 +1,74 @@ +package push + +import ( + "io" + "log/slog" + "testing" + "time" +) + +// restart is the same registry under a new service, as after a deploy. +func (r *rig) restart(t *testing.T) { + t.Helper() + r.svc = New(r.reg, r.apple, slog.New(slog.NewTextHandler(io.Discard, nil))) + r.svc.now, r.svc.after = r.clock.time, r.clock.after + if err := r.svc.Resume(); err != nil { + t.Fatal(err) + } +} + +// A restart in the middle of an outage keeps what was left of the wait. +func TestARestartKeepsTheWaitForBoxUnreachable(t *testing.T) { + r := newRig(t) + _ = r.register(t, r.owner, r.all, "tok") + r.svc.BoxConnected(r.home) + r.svc.BoxGone(r.home, "") + r.clock.advance(400 * time.Second) + r.restart(t) + r.clock.advance(199 * time.Second) + if n := unreachableSent(t, r); n != 0 { + t.Fatal("sent before ten minutes") + } + r.clock.advance(2 * time.Second) + if n := unreachableSent(t, r); n != 1 { + t.Fatalf("sent %d at ten minutes, across a restart", n) + } +} + +// An alert already sent is not sent again by a restart. +func TestARestartDoesNotRepeatBoxUnreachable(t *testing.T) { + r := newRig(t) + _ = r.register(t, r.owner, r.all, "tok") + r.svc.BoxConnected(r.home) + r.svc.BoxGone(r.home, "") + r.clock.advance(601 * time.Second) + if n := unreachableSent(t, r); n != 1 { + t.Fatalf("sent %d", n) + } + r.restart(t) + r.clock.advance(time.Hour) + if n := unreachableSent(t, r); n != 1 { + t.Fatalf("sent %d after a restart", n) + } +} + +// A box that was connected when the service stopped gets the whole wait to +// come back, and nothing is sent when it does. +func TestARestartGivesAConnectedBoxItsWholeWait(t *testing.T) { + r := newRig(t) + _ = r.register(t, r.owner, r.all, "tok") + r.svc.BoxConnected(r.home) + r.restart(t) + r.clock.advance(30 * time.Second) + r.svc.BoxConnected(r.home) + r.clock.advance(time.Hour) + if n := unreachableSent(t, r); n != 0 { + t.Fatalf("sent %d for a box that came back", n) + } + + r.restart(t) + r.clock.advance(601 * time.Second) + if n := unreachableSent(t, r); n != 1 { + t.Fatalf("sent %d for a box that never came back after a restart", n) + } +} diff --git a/internal/registry/push.go b/internal/registry/push.go index 9b188ca..790942d 100644 --- a/internal/registry/push.go +++ b/internal/registry/push.go @@ -59,6 +59,24 @@ CREATE TABLE IF NOT EXISTS push ( return nil } +// PushHomes lists the homes with a registration of a kind. +func (r *Registry) PushHomes(kind string) ([]string, error) { + rows, err := r.db.Query(`SELECT DISTINCT home FROM push WHERE kind = ?`, kind) + if err != nil { + return nil, err + } + defer rows.Close() + var out []string + for rows.Next() { + var h string + if err := rows.Scan(&h); err != nil { + return nil, err + } + out = append(out, h) + } + return out, rows.Err() +} + // PushRegister keeps a device's token. A token already registered for the // home moves to this device: a phone that paired again keeps its token. func (r *Registry) PushRegister(reg PushReg) error { diff --git a/internal/registry/registry.go b/internal/registry/registry.go index 056f8db..2c691a1 100644 --- a/internal/registry/registry.go +++ b/internal/registry/registry.go @@ -71,6 +71,11 @@ CREATE TABLE IF NOT EXISTS suspensions ( epoch INTEGER NOT NULL, since_ms INTEGER NOT NULL, PRIMARY KEY (home, device_id) +); +CREATE TABLE IF NOT EXISTS away ( + home TEXT PRIMARY KEY REFERENCES homes(home) ON DELETE CASCADE, + due_ms INTEGER NOT NULL, + told INTEGER NOT NULL DEFAULT 0 );`) if err != nil { _ = db.Close() @@ -128,6 +133,75 @@ func (r *Registry) Get(home string) (Home, bool, error) { return h, true, nil } +// Rosters returns every home that holds a signed roster, for the service to +// admit phones by again after a restart. It is upkeep at start, never a +// request. +func (r *Registry) Rosters() ([]Home, error) { + rows, err := r.db.Query(`SELECT home, pk, created_ms, roster, roster_seq, roster_ms, grant_id FROM homes WHERE roster IS NOT NULL`) + if err != nil { + return nil, err + } + defer rows.Close() + var out []Home + for rows.Next() { + var h Home + var created, seq, rosterMs int64 + if err := rows.Scan(&h.ID, &h.PK, &created, &h.Roster, &seq, &rosterMs, &h.GrantID); err != nil { + return nil, err + } + h.CreatedAt, h.RosterSeq, h.RosterAt = time.UnixMilli(created), uint64(seq), time.UnixMilli(rosterMs) + out = append(out, h) + } + return out, rows.Err() +} + +// Away is a home whose box is gone, as the dead man's switch keeps it across +// a restart: when box.unreachable is due, and whether it went. +type Away struct { + Home string + Due time.Time + Told bool +} + +// AwaySet records that a home's box went and when box.unreachable is due. +func (r *Registry) AwaySet(home string, due time.Time) error { + _, err := r.db.Exec(`INSERT INTO away (home, due_ms, told) SELECT ?, ?, 0 WHERE EXISTS (SELECT 1 FROM homes WHERE home = ?) + ON CONFLICT (home) DO UPDATE SET due_ms = excluded.due_ms, told = 0`, home, due.UnixMilli(), home) + return err +} + +// AwayTold records that box.unreachable went. +func (r *Registry) AwayTold(home string) error { + _, err := r.db.Exec(`UPDATE away SET told = 1 WHERE home = ?`, home) + return err +} + +// AwayClear forgets a home's absence: its box is back, or the home is gone. +func (r *Registry) AwayClear(home string) error { + _, err := r.db.Exec(`DELETE FROM away WHERE home = ?`, home) + return err +} + +// AwayAll returns every home recorded as away. +func (r *Registry) AwayAll() ([]Away, error) { + rows, err := r.db.Query(`SELECT home, due_ms, told FROM away`) + if err != nil { + return nil, err + } + defer rows.Close() + var out []Away + for rows.Next() { + var a Away + var due int64 + if err := rows.Scan(&a.Home, &due, &a.Told); err != nil { + return nil, err + } + a.Due = time.UnixMilli(due) + out = append(out, a) + } + return out, rows.Err() +} + // SetRoster keeps the signed roster envelope. func (r *Registry) SetRoster(home string, seq uint64, grantID, envelope []byte, now time.Time) error { _, err := r.db.Exec(`UPDATE homes SET roster = ?, roster_seq = ?, roster_ms = ?, grant_id = ? WHERE home = ? AND roster_seq <= ?`, diff --git a/internal/server/e2e_test.go b/internal/server/e2e_test.go index 965ee84..2f590fe 100644 --- a/internal/server/e2e_test.go +++ b/internal/server/e2e_test.go @@ -62,6 +62,8 @@ type stack struct { serviceKey *ecdsa.PrivateKey noise appwire.KeyPair apple *capturedApple + up *uplink.Uplink + reg *registry.Registry } func newStack(t *testing.T) *stack { @@ -82,7 +84,7 @@ func newStack(t *testing.T) *stack { pushSvc.WatchApp = "energy.ftw.app.watchkitapp" tokens := token.New(st, reg, log) app := appsession.New(st, reg, up, noise, log) - app.Push, app.Tokens = pushSvc, tokens + app.Push, app.Tokens, app.Tunnel = pushSvc, tokens, up up.OnHeld = pushSvc.Held up.OnRoster = func(home string, r *cloudwire.Roster) { pushSvc.Roster(home, r) @@ -102,7 +104,7 @@ func newStack(t *testing.T) *stack { srv.TLS = &tls.Config{MinVersion: tls.VersionTLS13} srv.StartTLS() t.Cleanup(srv.Close) - return &stack{t: t, srv: srv, st: st, serviceKey: serviceKey, noise: noise, apple: apple} + return &stack{t: t, srv: srv, st: st, serviceKey: serviceKey, noise: noise, apple: apple, up: up, reg: reg} } // testBox is a box's half of the uplink. diff --git a/internal/server/tunnel_e2e_test.go b/internal/server/tunnel_e2e_test.go new file mode 100644 index 0000000..beac4ce --- /dev/null +++ b/internal/server/tunnel_e2e_test.go @@ -0,0 +1,700 @@ +package server + +import ( + "bytes" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "net/http" + "sort" + "strings" + "sync" + "testing" + "time" + + "github.com/gorilla/websocket" + "github.com/srcfl/ftw/go/protocol" + "github.com/srcfl/ftw/go/protocol/appwire" + "github.com/srcfl/ftw/go/protocol/cloudwire" +) + +// The tunnel through the whole service, as ftwcloudd wires it: a box that +// takes tun, phones on /v1/app, and each phone's own Noise session with the +// box riding inside, which the service routes and cannot read. + +// sessionPrologue is the box session's prologue, spelled here on its own as +// a check on the box's spelling (appuplink.prologue in srcfl/ftw). +func sessionPrologue(boxStatic []byte) []byte { + return append([]byte("ftw.session.v1:"), boxStatic...) +} + +// tunBox is a box that links for its phones: caps with tun, a roster, and +// the box's half of each phone's session. +type tunBox struct { + t *testing.T + key *ecdsa.PrivateKey + home string + static appwire.KeyPair + conn *websocket.Conn + wmu sync.Mutex + seq uint64 + + mu sync.Mutex + allowed map[string][]byte // device id to key, from the box's roster + code []byte + sessions map[string]*boxSide + gotDevs []string + ends []cloudwire.Tun + heard []string + purged chan cloudwire.Purged + closed chan struct{} +} + +type boxSide struct { + sid uint32 + tr *appwire.Transport +} + +func (s *stack) connectTunBox() *tunBox { + s.t.Helper() + key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + pk, _ := cloudwire.PublicKeyBytesOf(&key.PublicKey) + static, _ := appwire.GenerateKeyPair() + b := &tunBox{t: s.t, key: key, home: cloudwire.HomeID(pk), static: static, allowed: map[string][]byte{}, + sessions: map[string]*boxSide{}, purged: make(chan cloudwire.Purged, 4), closed: make(chan struct{})} + pool := x509.NewCertPool() + pool.AddCert(s.srv.Certificate()) + d := websocket.Dialer{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS13}} + conn, _, err := d.Dial(strings.Replace(s.srv.URL, "https://", "wss://", 1)+"/v1/box", nil) + if err != nil { + s.t.Fatal(err) + } + s.t.Cleanup(func() { _ = conn.Close() }) + b.conn = conn + cs := conn.NetConn().(*tls.Conn).ConnectionState() + ekm, _ := cs.ExportKeyingMaterial(cloudwire.EKMLabel, nil, cloudwire.EKMBytes) + var ch cloudwire.Challenge + if env := b.read(); env.T != cloudwire.MsgChallenge || cloudwire.Unmarshal(env.B, &ch) != nil { + s.t.Fatalf("first message %q", env.T) + } + b.send(cloudwire.MsgHello, cloudwire.Hello{Header: b.header(), V: cloudwire.Version, PK: pk, Nonce: ch.Nonce, + EKM: ekm, BoxNonce: bytes.Repeat([]byte{4}, cloudwire.NonceBytes), Build: "e2e", Clock: cloudwire.ClockNTP, + Caps: []string{cloudwire.MsgRoster, cloudwire.CapTun}, RosterSeq: 1}) + if env := b.read(); env.T != cloudwire.MsgHelloOK { + s.t.Fatalf("second message %q, want hello_ok", env.T) + } + go b.serve() + return b +} + +func (b *tunBox) header() cloudwire.Header { + b.seq++ + return cloudwire.Header{Seq: b.seq, Boot: bytes.Repeat([]byte{0xb1}, cloudwire.BootBytes), UptimeMs: int64(b.seq)} +} + +func (b *tunBox) read() cloudwire.Envelope { + b.t.Helper() + _ = b.conn.SetReadDeadline(time.Now().Add(3 * time.Second)) + _, msg, err := b.conn.ReadMessage() + if err != nil { + b.t.Fatal(err) + } + payload, _ := cloudwire.DecodeFrame(msg) + env, err := cloudwire.DecodeEnvelope(payload) + if err != nil { + b.t.Fatal(err) + } + return env +} + +func (b *tunBox) write(env cloudwire.Envelope) { + raw, _ := env.Encode() + frame, _ := cloudwire.EncodeFrame(raw) + b.wmu.Lock() + defer b.wmu.Unlock() + _ = b.conn.WriteMessage(websocket.BinaryMessage, frame) +} + +func (b *tunBox) send(t string, body any) { + env, err := cloudwire.Seal(b.key, b.home, t, body) + if err != nil { + b.t.Fatal(err) + } + b.write(env) +} + +func (b *tunBox) tun(t cloudwire.Tun) { + body, _ := cloudwire.Marshal(t) + b.write(cloudwire.Envelope{T: cloudwire.MsgTun, B: body}) +} + +// roster lists phones that may open a tunnel, the readable copy off, and +// the hash of a pairing code when one is on the screen. +func (b *tunBox) roster(seq uint64, grant cloudwire.Grant, code []byte, keys ...[]byte) { + b.mu.Lock() + b.code = code + r := cloudwire.Roster{Header: b.header(), RosterSeq: seq, TTLS: 86_400, Grant: grant} + for _, k := range keys { + id := cloudwire.DeviceID(k) + b.allowed[id] = k + r.Devices = append(r.Devices, cloudwire.Device{ID: id, Key: k, Role: "owner", Scopes: protocol.CloudScopes, Epoch: 1, Tun: true}) + } + if code != nil { + r.Pending = []cloudwire.Pending{{Hash: cloudwire.PairingHash(b.home, code), TTLS: 600}} + } + b.mu.Unlock() + sort.Slice(r.Devices, func(i, j int) bool { return r.Devices[i].ID < r.Devices[j].ID }) + b.send(cloudwire.MsgRoster, r) +} + +var grantOff = cloudwire.Grant{Scopes: []string{}, Purposes: []string{}, Controller: cloudwire.ControllerSourceful} + +// serve is the box: it answers each phone's handshake for a key its roster +// lists, or with the code on its screen, and answers every cmd with an ack +// and a result. +func (b *tunBox) serve() { + defer close(b.closed) + for { + _ = b.conn.SetReadDeadline(time.Time{}) + _, msg, err := b.conn.ReadMessage() + if err != nil { + return + } + payload, _ := cloudwire.DecodeFrame(msg) + env, err := cloudwire.DecodeEnvelope(payload) + if err != nil { + b.t.Errorf("service frame: %v", err) + return + } + b.mu.Lock() + b.heard = append(b.heard, env.T) + b.mu.Unlock() + switch env.T { + case cloudwire.MsgPurged: + var p cloudwire.Purged + _ = cloudwire.Unmarshal(env.B, &p) + b.purged <- p + case cloudwire.MsgTun: + var t cloudwire.Tun + if len(env.Sig) != 0 || cloudwire.Unmarshal(env.B, &t) != nil { + b.t.Error("a tun the box cannot read") + return + } + b.inner(t) + } + } +} + +func (b *tunBox) inner(t cloudwire.Tun) { + b.mu.Lock() + b.gotDevs = append(b.gotDevs, t.Dev) + if t.Ended() { + b.ends = append(b.ends, t) + delete(b.sessions, t.Dev) + b.mu.Unlock() + return + } + side := b.sessions[t.Dev] + b.mu.Unlock() + if side != nil && side.sid == t.Sid { + frame, err := side.tr.Decrypt(t.D) + if err != nil { + return + } + f, err := appwire.DecodeFrame(frame) + if err != nil || f.Envelope.T != protocol.MsgCmd { + return + } + var cmd protocol.Cmd + _ = protocol.Unmarshal(f.Envelope.B, &cmd) + for _, out := range []struct { + t string + body any + }{ + {protocol.MsgCmdAck, protocol.CmdAck{CmdID: cmd.CmdID, LeaseID: "lease-1", ExpiresAtMs: 60_000}}, + {protocol.MsgCmdResult, protocol.CmdResult{CmdID: cmd.CmdID, State: protocol.CmdApplied}}, + } { + raw, _ := appwire.MarshalBody(out.body) + reply, _ := appwire.EncodeFrame(appwire.Frame{Lane: appwire.LaneControl, Bucket: 512, + Envelope: appwire.Envelope{T: out.t, B: raw}}) + d, _ := side.tr.Encrypt(reply) + b.tun(cloudwire.Tun{Dev: t.Dev, Sid: t.Sid, D: d}) + } + return + } + hs, err := appwire.NewResponder(appwire.HandshakeOptions{StaticKey: b.static, Prologue: sessionPrologue(b.static.Public())}) + if err != nil { + return + } + code, err := hs.ReadMessage(t.D) + if err != nil { + b.tun(cloudwire.Tun{Dev: t.Dev, Sid: t.Sid}) + return + } + b.mu.Lock() + known := bytes.Equal(b.allowed[t.Dev], hs.RemoteStatic()) + paired := len(code) > 0 && bytes.Equal(code, b.code) && cloudwire.DeviceID(hs.RemoteStatic()) == t.Dev + b.mu.Unlock() + if !known && !paired { + return + } + reply, _ := hs.WriteMessage(nil) + keys, _ := hs.Split() + b.mu.Lock() + b.sessions[t.Dev] = &boxSide{sid: t.Sid, tr: appwire.NewTransport(keys)} + b.mu.Unlock() + b.tun(cloudwire.Tun{Dev: t.Dev, Sid: t.Sid, D: reply}) +} + +func (b *tunBox) devsSeen() []string { + b.mu.Lock() + defer b.mu.Unlock() + return append([]string(nil), b.gotDevs...) +} + +func (b *tunBox) endsSeen() []cloudwire.Tun { + b.mu.Lock() + defer b.mu.Unlock() + return append([]cloudwire.Tun(nil), b.ends...) +} + +// tunPhone is a phone: its app session with the service, and inside it its +// own session with the box. +type tunPhone struct { + t *testing.T + key appwire.KeyPair + conn *websocket.Conn + tr *appwire.Transport + snap cloudwire.Snapshot + inner *appwire.Transport + in *appwire.InitiatorState + tuns chan cloudwire.AppTun + code int +} + +// dialPhone opens the app session. A refusal comes back as code, before +// message 2. +func (s *stack) dialPhone(home string, key appwire.KeyPair, now cloudwire.Now, origin string) *tunPhone { + s.t.Helper() + p := &tunPhone{t: s.t, key: key, tuns: make(chan cloudwire.AppTun, 64)} + pool := x509.NewCertPool() + pool.AddCert(s.srv.Certificate()) + d := websocket.Dialer{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS13}} + h := http.Header{} + if origin != "" { + h.Set("Origin", origin) + } + conn, resp, err := d.Dial(strings.Replace(s.srv.URL, "https://", "wss://", 1)+"/v1/app?home="+home, h) + if err != nil { + if resp != nil { + p.code = resp.StatusCode + } + return p + } + s.t.Cleanup(func() { _ = conn.Close() }) + p.conn = conn + init, err := appwire.NewInitiator(appwire.InitiatorOptions{StaticKey: key, RemoteStatic: s.noise.Public(), + Prologue: cloudwire.AppPrologue(home)}) + if err != nil { + s.t.Fatal(err) + } + payload, _ := cloudwire.Unsigned(cloudwire.MsgNow, now) + msg1, _ := init.WriteMessage(payload) + _ = conn.WriteMessage(websocket.BinaryMessage, msg1) + _ = conn.SetReadDeadline(time.Now().Add(3 * time.Second)) + _, msg2, err := conn.ReadMessage() + if err != nil { + var ce *websocket.CloseError + if ok := asClose(err, &ce); ok { + p.code = ce.Code + } + return p + } + reply, err := init.ReadMessage(msg2) + if err != nil { + s.t.Fatal(err) + } + if p.snap, err = cloudwire.ParseSnapshot(reply); err != nil { + s.t.Fatal(err) + } + keys, _ := init.Split() + p.tr = appwire.NewTransport(keys) + go func() { + for { + _ = conn.SetReadDeadline(time.Time{}) + _, msg, err := conn.ReadMessage() + if err != nil { + close(p.tuns) + return + } + frame, err := p.tr.Decrypt(msg) + if err != nil { + continue + } + payload, _ := cloudwire.DecodeFrame(frame) + env, _ := cloudwire.DecodeEnvelope(payload) + if env.T == cloudwire.MsgTun { + var t cloudwire.AppTun + _ = cloudwire.Unmarshal(env.B, &t) + p.tuns <- t + } + } + }() + return p +} + +func asClose(err error, ce **websocket.CloseError) bool { + c, ok := err.(*websocket.CloseError) + *ce = c + return ok +} + +// sendRaw sends any body as a tun envelope, as a phone might shape it. +func (p *tunPhone) sendRaw(body any) { + p.t.Helper() + env, _ := cloudwire.Unsigned(cloudwire.MsgTun, body) + frame, _ := cloudwire.EncodeFrame(env) + msg, _ := p.tr.Encrypt(frame) + if err := p.conn.WriteMessage(websocket.BinaryMessage, msg); err != nil { + p.t.Fatal(err) + } +} + +func (p *tunPhone) next() cloudwire.AppTun { + p.t.Helper() + select { + case t, ok := <-p.tuns: + if !ok { + p.t.Fatal("the app session closed") + } + return t + case <-time.After(3 * time.Second): + p.t.Fatal("no tun reached the phone") + return cloudwire.AppTun{} + } +} + +// boxMessage1 starts the phone's session with the box. +func (p *tunPhone) boxMessage1(boxStatic, code []byte) []byte { + p.t.Helper() + in, err := appwire.NewInitiator(appwire.InitiatorOptions{StaticKey: p.key, RemoteStatic: boxStatic, + Prologue: sessionPrologue(boxStatic)}) + if err != nil { + p.t.Fatal(err) + } + p.in = in + m1, _ := in.WriteMessage(code) + return m1 +} + +func (p *tunPhone) boxMessage2(d []byte) { + p.t.Helper() + if _, err := p.in.ReadMessage(d); err != nil { + p.t.Fatal(err) + } + keys, _ := p.in.Split() + p.inner = appwire.NewTransport(keys) +} + +// open runs the box session through the tunnel. +func (p *tunPhone) open(boxStatic []byte, sid uint32, code []byte) { + p.t.Helper() + p.sendRaw(cloudwire.AppTun{Sid: sid, D: p.boxMessage1(boxStatic, code)}) + reply := p.next() + if reply.Sid != sid || len(reply.D) == 0 { + p.t.Fatalf("the box answered %+v", reply) + } + p.boxMessage2(reply.D) +} + +// cmd sends a command inside the box session and returns the states of the +// ack and result. +func (p *tunPhone) cmd(sid uint32, id string) (string, string) { + p.t.Helper() + raw, _ := appwire.MarshalBody(protocol.Cmd{CmdID: id, Op: "mode.set", Args: map[string]any{"mode": "self_consumption"}, + NotValidAfterMs: 60_000, Expect: protocol.Expect{Rev: 1}}) + frame, _ := appwire.EncodeFrame(appwire.Frame{Lane: appwire.LaneControl, Bucket: 512, + Envelope: appwire.Envelope{T: protocol.MsgCmd, B: raw}}) + d, _ := p.inner.Encrypt(frame) + p.sendRaw(cloudwire.AppTun{Sid: sid, D: d}) + var ack, res string + for ack == "" || res == "" { + t := p.next() + plain, err := p.inner.Decrypt(t.D) + if err != nil { + p.t.Fatal(err) + } + f, _ := appwire.DecodeFrame(plain) + switch f.Envelope.T { + case protocol.MsgCmdAck: + ack = f.Envelope.T + case protocol.MsgCmdResult: + var r protocol.CmdResult + _ = protocol.Unmarshal(f.Envelope.B, &r) + res = r.State + } + } + return ack, res +} + +func eventually(t *testing.T, what string, cond func() bool) { + t.Helper() + deadline := time.Now().Add(3 * time.Second) + for !cond() { + if time.Now().After(deadline) { + t.Fatalf("never: %s", what) + } + time.Sleep(10 * time.Millisecond) + } +} + +// A phone the roster lists sends a command through the service and gets the +// box's answer, with the readable copy off. Nothing to read is held. +func TestACommandThroughTheServiceGetsTheBoxsAnswer(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() + key, _ := appwire.GenerateKeyPair() + box.roster(2, grantOff, nil, key.Public()) + // A reading the box sends anyway is not kept: the copy is off. + box.send(cloudwire.MsgLive, cloudwire.Live{Header: box.header(), Fields: map[string]int64{"2": 1}, Ages: map[string]int64{}}) + + var phone *tunPhone + eventually(t, "the phone is admitted", func() bool { + phone = s.dialPhone(box.home, key, cloudwire.Now{Hz: 1}, "") + return phone.conn != nil && phone.tr != nil + }) + if len(phone.snap.Envelopes) != 0 { + t.Fatalf("the snapshot holds %d envelopes with the copy off", len(phone.snap.Envelopes)) + } + phone.open(box.static.Public(), 7, nil) + + var took []time.Duration + for i := range 20 { + start := time.Now() + ack, res := phone.cmd(7, "0192f2a0-7c1e-7000-8000-0000000000"+string(rune('a'+i))+"0") + took = append(took, time.Since(start)) + if ack != protocol.MsgCmdAck || res != protocol.CmdApplied { + t.Fatalf("ack %q, result %q", ack, res) + } + } + sort.Slice(took, func(a, b int) bool { return took[a] < took[b] }) + t.Logf("cmd to result through ftwcloudd, 20 runs: median %s, max %s", took[10], took[19]) + for _, d := range box.devsSeen() { + if d != cloudwire.DeviceID(key.Public()) { + t.Fatalf("the box got a tun under %q", d) + } + } +} + +// The box session's message 1 may ride in the app session's message 1, and +// the box's answer follows message 2. +func TestTheBoxSessionRidesInMessage1(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() + key, _ := appwire.GenerateKeyPair() + box.roster(2, grantOff, nil, key.Public()) + waitRoster(t, s, box.home, key) + p := &tunPhone{t: t, key: key} + m1 := p.boxMessage1(box.static.Public(), nil) + phone := s.dialPhone(box.home, key, cloudwire.Now{Hz: 1, Tun: &cloudwire.AppTun{Sid: 3, D: m1}}, "") + if phone.tr == nil { + t.Fatalf("refused with %d", phone.code) + } + phone.in = p.in + reply := phone.next() + if reply.Sid != 3 || len(reply.D) == 0 { + t.Fatalf("the box answered %+v", reply) + } + phone.boxMessage2(reply.D) + if ack, res := phone.cmd(3, "0192f2a0-7c1e-7000-8000-00000000e001"); ack == "" || res != protocol.CmdApplied { + t.Fatalf("ack %q result %q", ack, res) + } +} + +func waitRoster(t *testing.T, s *stack, home string, key appwire.KeyPair) { + t.Helper() + eventually(t, "the roster lists the phone", func() bool { + p := s.dialPhone(home, key, cloudwire.Now{}, "") + if p.conn != nil { + _ = p.conn.Close() + } + return p.tr != nil + }) +} + +// A key the roster does not list is refused before message 2, and nothing +// it sent reaches the box. +func TestAPhoneNotInTheRosterIsRefusedBeforeAnythingIsForwarded(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() + known, _ := appwire.GenerateKeyPair() + box.roster(2, grantOff, nil, known.Public()) + waitRoster(t, s, box.home, known) + + stranger, _ := appwire.GenerateKeyPair() + p := &tunPhone{t: t, key: stranger} + m1 := p.boxMessage1(box.static.Public(), nil) + phone := s.dialPhone(box.home, stranger, cloudwire.Now{Hz: 1, Tun: &cloudwire.AppTun{Sid: 1, D: m1}}, "") + if phone.tr != nil || phone.code != cloudwire.CloseDenied { + t.Fatalf("a stranger got a session (code %d)", phone.code) + } + time.Sleep(200 * time.Millisecond) + if devs := box.devsSeen(); len(devs) != 0 { + t.Fatalf("the box got tun from %v", devs) + } +} + +// The service names the device by the key its own handshake proved. A dev a +// phone puts in its tun is ignored. +func TestAForgedDevIsIgnored(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() + mine, _ := appwire.GenerateKeyPair() + other, _ := appwire.GenerateKeyPair() + box.roster(2, grantOff, nil, mine.Public(), other.Public()) + waitRoster(t, s, box.home, mine) + phone := s.dialPhone(box.home, mine, cloudwire.Now{Hz: 1}, "") + phone.sendRaw(map[string]any{"dev": cloudwire.DeviceID(other.Public()), "sid": 5, + "d": phone.boxMessage1(box.static.Public(), nil)}) + reply := phone.next() + if reply.Sid != 5 || len(reply.D) == 0 { + t.Fatalf("the box answered %+v", reply) + } + for _, d := range box.devsSeen() { + if d != cloudwire.DeviceID(mine.Public()) { + t.Fatalf("the box got a tun under %q, the forged id", d) + } + } +} + +// With the readable copy withdrawn, the service drops what it held, signs a +// receipt for the grant, and the tunnel still works. +func TestTheCopyStaysOffWhileTheTunnelWorks(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() + key, _ := appwire.GenerateKeyPair() + grantID := bytes.Repeat([]byte{7}, cloudwire.GrantIDBytes) + on := cloudwire.Grant{ID: grantID, Scopes: []string{protocol.ScopeLiveRead}, + Purposes: []string{protocol.CloudPurposeServe}, Controller: cloudwire.ControllerSourceful} + box.roster(2, on, nil, key.Public()) + box.send(cloudwire.MsgLive, cloudwire.Live{Header: box.header(), Fields: map[string]int64{"2": 1}, Ages: map[string]int64{}}) + eventually(t, "the copy is held", func() bool { + p := s.dialPhone(box.home, key, cloudwire.Now{}, "") + if p.conn != nil { + _ = p.conn.Close() + } + return len(p.snap.Envelopes) == 1 + }) + + withdrawn := cloudwire.Grant{ID: grantID, Scopes: []string{}, Purposes: []string{}, Controller: cloudwire.ControllerSourceful} + box.roster(3, withdrawn, nil, key.Public()) + select { + case p := <-box.purged: + if p.What != cloudwire.PurgedCopy || !bytes.Equal(p.GrantID, grantID) { + t.Fatalf("receipt %+v", p) + } + case <-time.After(3 * time.Second): + t.Fatal("no receipt for the withdrawn copy") + } + box.send(cloudwire.MsgLive, cloudwire.Live{Header: box.header(), Fields: map[string]int64{"2": 2}, Ages: map[string]int64{}}) + phone := s.dialPhone(box.home, key, cloudwire.Now{Hz: 1}, "") + if phone.tr == nil || len(phone.snap.Envelopes) != 0 { + t.Fatalf("after the receipt: code %d, %d envelopes", phone.code, len(phone.snap.Envelopes)) + } + phone.open(box.static.Public(), 1, nil) + if _, res := phone.cmd(1, "0192f2a0-7c1e-7000-8000-00000000f001"); res != protocol.CmdApplied { + t.Fatalf("result %q", res) + } +} + +// A new phone pairs through the service: the roster lists the code's hash, +// the phone proves it once, and its session stays open when the roster +// lists it. +func TestAPhonePairsThroughTheService(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() + owner, _ := appwire.GenerateKeyPair() + code := bytes.Repeat([]byte{0x42}, 16) + box.roster(2, grantOff, code, owner.Public()) + waitRoster(t, s, box.home, owner) + + newKey, _ := appwire.GenerateKeyPair() + if p := s.dialPhone(box.home, newKey, cloudwire.Now{Hz: 1}, ""); p.tr != nil { + t.Fatal("a new phone without the code was admitted") + } + if p := s.dialPhone(box.home, newKey, cloudwire.Now{Pair: cloudwire.PairingHash(box.home, []byte("wrong-code-000000"))}, ""); p.tr != nil { + t.Fatal("a wrong pairing hash was admitted") + } + phone := s.dialPhone(box.home, newKey, cloudwire.Now{Hz: 1, Pair: cloudwire.PairingHash(box.home, code)}, "") + if phone.tr == nil { + t.Fatalf("the pairing phone was refused with %d", phone.code) + } + phone.open(box.static.Public(), 1, code) + if devs := box.devsSeen(); devs[len(devs)-1] != cloudwire.DeviceID(newKey.Public()) { + t.Fatalf("the box got the pairing under %v", devs) + } + // The hash admits one handshake. + again, _ := appwire.GenerateKeyPair() + if p := s.dialPhone(box.home, again, cloudwire.Now{Pair: cloudwire.PairingHash(box.home, code)}, ""); p.tr != nil { + t.Fatal("a pairing hash admitted a second phone") + } + // The box paired it: the roster lists it, and the session stays. + box.roster(3, grantOff, nil, owner.Public(), newKey.Public()) + time.Sleep(200 * time.Millisecond) + if _, res := phone.cmd(1, "0192f2a0-7c1e-7000-8000-00000000a0a1"); res != protocol.CmdApplied { + t.Fatalf("result %q after the roster listed the phone", res) + } +} + +// The ends: a phone that leaves ends its box session at the box, and a box +// that goes ends the phone's. +func TestEachEndHearsTheOtherGo(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() + a, _ := appwire.GenerateKeyPair() + b, _ := appwire.GenerateKeyPair() + box.roster(2, grantOff, nil, a.Public(), b.Public()) + waitRoster(t, s, box.home, a) + + leaving := s.dialPhone(box.home, a, cloudwire.Now{Hz: 1}, "") + leaving.open(box.static.Public(), 4, nil) + _ = leaving.conn.Close() + eventually(t, "the box hears the phone left", func() bool { + for _, e := range box.endsSeen() { + if e.Dev == cloudwire.DeviceID(a.Public()) && e.Sid == 4 { + return true + } + } + return false + }) + + staying := s.dialPhone(box.home, b, cloudwire.Now{Hz: 1}, "") + staying.open(box.static.Public(), 9, nil) + _ = box.conn.Close() + end := staying.next() + if end.Sid != 9 || len(end.D) != 0 { + t.Fatalf("after the box went the phone got %+v", end) + } + // With no box, a new session is ended at once. + staying.sendRaw(cloudwire.AppTun{Sid: 10, D: staying.boxMessage1(box.static.Public(), nil)}) + if end := staying.next(); end.Sid != 10 || len(end.D) != 0 { + t.Fatalf("with no box the phone got %+v", end) + } +} + +// Browsers: the web app's origin is let in, any other is refused. +func TestOnlyTheWebAppsOriginIsLetIn(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() + key, _ := appwire.GenerateKeyPair() + box.roster(2, grantOff, nil, key.Public()) + waitRoster(t, s, box.home, key) + if p := s.dialPhone(box.home, key, cloudwire.Now{}, "https://evil.example"); p.conn != nil { + t.Fatal("another origin was let in") + } + if p := s.dialPhone(box.home, key, cloudwire.Now{}, "https://app.ftw.energy"); p.tr == nil { + t.Fatalf("the web app's origin was refused: %d", p.code) + } +} diff --git a/internal/store/session.go b/internal/store/session.go index 8b8f657..fdae437 100644 --- a/internal/store/session.go +++ b/internal/store/session.go @@ -17,6 +17,11 @@ const ( // dead socket. It is closed and gets a fresh snapshot when it comes // back, rather than a stream with a hole in it. SessionQueue = 256 + // TunQueue is how many tun messages may wait for one phone, apart from + // the envelopes, so a reading never waits behind the tunnel or the + // other way round. Past it a message is dropped, which Noise transport + // survives at the cost of one frame. + TunQueue = 256 // WebSocket close codes for the two endings that are not the protocol's // own: the phone left, or it fell too far behind. @@ -30,6 +35,11 @@ const ( type Session struct { x *Handle c chan []byte + // tun carries the box's side of the phone's own session. + tun chan cloudwire.AppTun + // sid is the box session the phone runs now, when hasSid. + sid uint32 + hasSid bool // done is closed when the store ends the session; code says why, as a // close code from cloudwire. done chan struct{} @@ -48,8 +58,9 @@ func (x *Handle) Open() (Snapshot, *Session, error) { x.h.mu.Lock() defer x.h.mu.Unlock() // The access checked in For may have changed since: check it again - // under the lock that roster changes take. - if _, _, ok := x.s.access(x.id, x.h, byKey(x.device.Key)); !ok { + // under the lock that roster changes take. A pairing was checked, and + // spent, there. + if _, _, ok := x.s.access(x.id, x.h, byKey(x.device.Key)); !ok && !x.pairing { return Snapshot{}, nil, ErrDenied } for sess := range x.h.sessions { @@ -60,7 +71,8 @@ func (x *Handle) Open() (Snapshot, *Session, error) { if len(x.h.sessions) >= MaxSessionsPerHome { return Snapshot{}, nil, ErrBusy } - sess := &Session{x: x, c: make(chan []byte, SessionQueue), done: make(chan struct{})} + sess := &Session{x: x, c: make(chan []byte, SessionQueue), tun: make(chan cloudwire.AppTun, TunQueue), + done: make(chan struct{})} x.h.sessions[sess] = struct{}{} return x.snapshotLocked(), sess, nil } @@ -68,6 +80,67 @@ func (x *Handle) Open() (Snapshot, *Session, error) { // C carries the envelopes to forward. func (s *Session) C() <-chan []byte { return s.c } +// Tun carries the box's tun messages for this phone. +func (s *Session) Tun() <-chan cloudwire.AppTun { return s.tun } + +// SetSid records the box session the phone runs now, so the end of the +// box's uplink reaches it. +func (s *Session) SetSid(sid uint32) { + s.x.h.mu.Lock() + defer s.x.h.mu.Unlock() + s.sid, s.hasSid = sid, true +} + +// Ended queues the end of the phone's box session sid, from the service +// itself, as when no box took its first message. +func (s *Session) Ended(sid uint32) { + s.x.h.mu.Lock() + defer s.x.h.mu.Unlock() + if s.ended { + return + } + select { + case s.tun <- cloudwire.AppTun{Sid: sid}: + default: + } +} + +// Tun hands one tun from the box to the session of the device it names. +// False when no session of that device is open. A full queue drops it. +func (x *Handle) Tun(t cloudwire.Tun) bool { + if x.kind != Box { + return false + } + x.h.mu.Lock() + defer x.h.mu.Unlock() + for sess := range x.h.sessions { + if sess.ended || sess.x.device.ID != t.Dev { + continue + } + select { + case sess.tun <- cloudwire.AppTun{Sid: t.Sid, D: t.D}: + default: + } + return true + } + return false +} + +// tunGoneLocked tells each phone whose box session rode the uplink that +// has gone that the session ended. The caller holds the home's lock. +func (h *home) tunGoneLocked() { + for sess := range h.sessions { + if !sess.hasSid || sess.ended { + continue + } + select { + case sess.tun <- cloudwire.AppTun{Sid: sess.sid}: + default: + } + sess.hasSid = false + } +} + // Done is closed when the store ends the session. func (s *Session) Done() <-chan struct{} { return s.done } @@ -108,14 +181,28 @@ func (s *Session) endLocked(code int) { close(s.done) } -// Device is the roster entry this device matched. -func (x *Handle) Device() cloudwire.Device { return x.device } +// Device is the roster entry this device matched. A phone admitted for +// pairing takes its entry when the roster lists it, so it is read under the +// home's lock. +func (x *Handle) Device() cloudwire.Device { + x.h.mu.Lock() + defer x.h.mu.Unlock() + return x.device +} // Scopes is the device's effective scope. -func (x *Handle) Scopes() []string { return x.scopes } +func (x *Handle) Scopes() []string { + x.h.mu.Lock() + defer x.h.mu.Unlock() + return x.scopes +} // Home is the home this handle reaches. func (x *Handle) Home() string { return x.id } // MayRead reports whether the device may read a class. -func (x *Handle) MayRead(class string) bool { return x.mayRead(class) } +func (x *Handle) MayRead(class string) bool { + x.h.mu.Lock() + defer x.h.mu.Unlock() + return x.mayRead(class) +} diff --git a/internal/store/store.go b/internal/store/store.go index 8e42d49..317eae1 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -11,12 +11,14 @@ package store import ( "bytes" + "encoding/hex" "errors" "slices" "sort" "sync" "time" + "github.com/srcfl/ftw/go/protocol" "github.com/srcfl/ftw/go/protocol/cloudwire" ) @@ -43,6 +45,9 @@ type Principal struct { Home string // DeviceKey is the phone's X25519 static key, for Device. DeviceKey []byte + // Pair is the pairing hash a new phone proved in its handshake, for a + // Device the roster does not list yet. It admits one session, once. + Pair []byte // DeviceID and Epoch are the roster entry a token is bound to, for // Token. DeviceID string @@ -92,6 +97,11 @@ type home struct { lastSeen time.Time // sessions are the phones reading this home now. sessions map[*Session]struct{} + // pending is each pairing hash the roster lists, by its hex, until it + // runs out; spent marks the ones a phone has used, so a roster sent + // again does not reopen them. + pending map[string]time.Time + spent map[string]time.Time } // Held is one envelope as received. @@ -111,6 +121,10 @@ type Handle struct { // effective scope: its roster scopes intersected with the grant's. device cloudwire.Device scopes []string + // pairing is a new phone admitted by a pairing hash until its pairing + // runs out at pairedUntil, or the roster lists its key. + pairing bool + pairedUntil time.Time } // For returns the principal's access to its home, or ErrDenied. A box may @@ -121,7 +135,7 @@ func (s *Store) For(p Principal) (*Handle, error) { s.mu.Lock() h := s.homes[p.Home] if h == nil && p.Kind == Box && p.Home != "" { - h = &home{latest: map[string]Held{}, sessions: map[*Session]struct{}{}} + h = newHome() s.homes[p.Home] = h } s.mu.Unlock() @@ -135,10 +149,14 @@ func (s *Store) For(p Principal) (*Handle, error) { h.mu.Lock() defer h.mu.Unlock() dev, scopes, ok := s.access(p.Home, h, byKey(p.DeviceKey)) - if !ok { - return nil, ErrDenied + if ok { + return &Handle{s: s, id: p.Home, h: h, kind: Device, device: dev, scopes: scopes}, nil + } + if until, ok := s.pair(h, p); ok { + return &Handle{s: s, id: p.Home, h: h, kind: Device, pairing: true, pairedUntil: until, + device: cloudwire.Device{ID: cloudwire.DeviceID(p.DeviceKey), Key: bytes.Clone(p.DeviceKey), Tun: true}}, nil } - return &Handle{s: s, id: p.Home, h: h, kind: Device, device: dev, scopes: scopes}, nil + return nil, ErrDenied case Token: h.mu.Lock() defer h.mu.Unlock() @@ -153,13 +171,40 @@ func (s *Store) For(p Principal) (*Handle, error) { return nil, ErrDenied } +func newHome() *home { + return &home{latest: map[string]Held{}, sessions: map[*Session]struct{}{}, + pending: map[string]time.Time{}, spent: map[string]time.Time{}} +} + +// pair spends a pending pairing hash for a key the roster does not list. The +// device it admits may only open a tunnel; the box decides inside it whether +// the phone pairs. The caller holds h.mu. +func (s *Store) pair(h *home, p Principal) (time.Time, bool) { + if len(p.Pair) != cloudwire.PairingHashBytes || len(p.DeviceKey) != 32 || h.roster == nil { + return time.Time{}, false + } + now := s.now() + if now.After(h.rosterAt.Add(time.Duration(h.roster.TTLS) * time.Second)) { + return time.Time{}, false + } + key := hex.EncodeToString(p.Pair) + until, ok := h.pending[key] + if !ok || !now.Before(until) { + return time.Time{}, false + } + delete(h.pending, key) + h.spent[key] = until + return until, true +} + // byKey matches the roster entry with a device's Noise static key. func byKey(key []byte) func(cloudwire.Device) bool { return func(d cloudwire.Device) bool { return len(key) == 32 && bytes.Equal(d.Key, key) } } // access is the one admission rule for a device: a roster in force that -// lists it, and no suspension. The caller holds h.mu. +// lists it, no suspension, and something the device may do here: open a +// tunnel, or read while the readable copy is on. The caller holds h.mu. func (s *Store) access(id string, h *home, match func(cloudwire.Device) bool) (cloudwire.Device, []string, bool) { r := h.roster if r == nil || s.now().After(h.rosterAt.Add(time.Duration(r.TTLS)*time.Second)) { @@ -178,6 +223,9 @@ func (s *Store) access(id string, h *home, match func(cloudwire.Device) bool) (c scopes = append(scopes, sc) } } + if !d.Tun && !slices.Contains(r.Grant.Purposes, protocol.CloudPurposeServe) { + return cloudwire.Device{}, nil, false + } return d, scopes, true } return cloudwire.Device{}, nil, false @@ -205,7 +253,28 @@ func (x *Handle) Disconnected(conn uint64) { if x.h.conn == conn { x.h.online = false x.h.lastSeen = x.s.now() + x.h.tunGoneLocked() + } +} + +// Serving reports whether the roster in force has the readable copy on. +// Without it the service holds no readings for the home. +func (x *Handle) Serving() bool { + x.h.mu.Lock() + defer x.h.mu.Unlock() + r := x.h.roster + return r != nil && slices.Contains(r.Grant.Purposes, protocol.CloudPurposeServe) +} + +// DropCopy forgets every reading the home holds, when its owner withdraws +// the readable copy. The home, its roster and its tunnel stay. +func (x *Handle) DropCopy() { + if x.kind != Box { + return } + x.h.mu.Lock() + defer x.h.mu.Unlock() + x.h.latest, x.h.full = map[string]Held{}, nil } // Put keeps env as the newest of its class. @@ -251,10 +320,35 @@ func (x *Handle) SetRoster(r *cloudwire.Roster) bool { return false } x.h.roster, x.h.rosterAt = r, x.s.now() + x.h.setPendingLocked(r, x.h.rosterAt) x.recheckLocked() return true } +// setPendingLocked takes the roster's pairing hashes, each until its ttlS +// runs out from when the roster arrived, and forgets the ones gone from it. +// A hash already spent stays spent. The caller holds h.mu. +func (h *home) setPendingLocked(r *cloudwire.Roster, at time.Time) { + listed := map[string]bool{} + for _, p := range r.Pending { + key := hex.EncodeToString(p.Hash) + listed[key] = true + if _, spent := h.spent[key]; !spent && len(p.Hash) == cloudwire.PairingHashBytes && p.TTLS > 0 { + h.pending[key] = at.Add(time.Duration(p.TTLS) * time.Second) + } + } + for key := range h.pending { + if !listed[key] { + delete(h.pending, key) + } + } + for key, until := range h.spent { + if at.After(until) { + delete(h.spent, key) + } + } +} + // Recheck ends every session whose access is no longer what it was opened // with, as after an owner suspends a device. The phone reconnects and learns // its new access, or is refused. @@ -268,6 +362,12 @@ func (x *Handle) recheckLocked() { for sess := range x.h.sessions { dev, scopes, ok := x.s.access(x.id, x.h, byKey(sess.x.device.Key)) switch { + case sess.x.pairing && ok: + // The box paired the phone: its session takes the roster's + // entry and stays open. + sess.x.device, sess.x.scopes, sess.x.pairing = dev, scopes, false + case sess.x.pairing && x.s.now().Before(sess.x.pairedUntil): + // Still pairing. case !ok: sess.endLocked(cloudwire.CloseDenied) case dev.ID != sess.x.device.ID || dev.Epoch != sess.x.device.Epoch || dev.Role != sess.x.device.Role || @@ -389,10 +489,36 @@ func (s *Store) Sweep() { sess.endLocked(cloudwire.CloseDenied) } } + // A pairing the box did not complete ends with its code. + for sess := range h.sessions { + if sess.x.pairing && !now.Before(sess.x.pairedUntil) { + sess.endLocked(cloudwire.CloseDenied) + } + } h.mu.Unlock() } } +// Seed holds a roster the box signed before the service started, read back +// from the registry, with the time it first arrived. It never replaces a +// roster the box sent since. Upkeep at start calls it, never a request. +func (s *Store) Seed(id string, r *cloudwire.Roster, at time.Time) { + s.mu.Lock() + h := s.homes[id] + if h == nil { + h = newHome() + s.homes[id] = h + } + s.mu.Unlock() + h.mu.Lock() + defer h.mu.Unlock() + if h.roster != nil { + return + } + h.roster, h.rosterAt, h.lastSeen = r, at, at + h.setPendingLocked(r, at) +} + // Forget drops a home the registry has deleted, as after 90 days without a // roster. It is called by upkeep, never by a request. func (s *Store) Forget(id string) { diff --git a/internal/store/store_test.go b/internal/store/store_test.go index 95b5bcb..20055da 100644 --- a/internal/store/store_test.go +++ b/internal/store/store_test.go @@ -26,7 +26,8 @@ func roster(seq uint64, grant ...string) *cloudwire.Roster { {ID: "owner", Key: ownerKey, Role: "owner", Scopes: protocol.CloudScopes}, {ID: "viewer", Key: viewerKey, Role: "viewer", Scopes: []string{protocol.ScopeLiveRead}}, }, - Grant: cloudwire.Grant{ID: make([]byte, 16), Scopes: grant, Controller: "sourceful"}} + Grant: cloudwire.Grant{ID: make([]byte, 16), Scopes: grant, Purposes: []string{protocol.CloudPurposeServe}, + Controller: "sourceful"}} } func filled(t *testing.T) (*Store, *Handle, *clock) { diff --git a/internal/uplink/downlink.go b/internal/uplink/downlink.go index 40dae17..69d32af 100644 --- a/internal/uplink/downlink.go +++ b/internal/uplink/downlink.go @@ -7,12 +7,14 @@ import ( ) // down is a message the service may send a box. The method is unexported, so -// only the five types below implement it and a sixth cannot be sent by +// only the six types below implement it and a seventh cannot be sent by // accident. TestDownlinkIsTheProtocolsClosedSet holds the list to // cloudwire.Downlink, and the box closes the uplink on anything else. // -// None of the five asks the box to act. watch asks it to send sooner, resync -// to send again, and purged confirms a deletion the box asked for. +// Five ask the box for nothing: watch asks it to send sooner, resync to send +// again, and purged confirms a deletion the box asked for. tun carries a +// phone's own Noise session, which the service cannot read or forge; the box +// judges it as it judges the same session over the relay. type down interface { down() (t string, body any, signed bool) } @@ -23,6 +25,7 @@ type ( watch cloudwire.Watch resync cloudwire.Resync purged cloudwire.Purged + tun cloudwire.Tun ) func (m challenge) down() (string, any, bool) { @@ -32,6 +35,7 @@ func (m helloOK) down() (string, any, bool) { return cloudwire.MsgHelloOK, cloud func (m watch) down() (string, any, bool) { return cloudwire.MsgWatch, cloudwire.Watch(m), false } func (m resync) down() (string, any, bool) { return cloudwire.MsgResync, cloudwire.Resync(m), false } func (m purged) down() (string, any, bool) { return cloudwire.MsgPurged, cloudwire.Purged(m), true } +func (m tun) down() (string, any, bool) { return cloudwire.MsgTun, cloudwire.Tun(m), false } // encodeDown frames one downlink message. hello_ok and purged are signed with // the service key over the box's home id; the rest are not. @@ -53,7 +57,11 @@ func (u *Uplink) encodeDown(home string, m down) ([]byte, error) { if err != nil { return nil, err } - if len(frame) > cloudwire.MaxServiceFrame { + limit := cloudwire.MaxServiceFrame + if t == cloudwire.MsgTun { + limit = cloudwire.MaxTunFrame + } + if len(frame) > limit { return nil, fmt.Errorf("uplink: %s is %d bytes, over the box's limit", t, len(frame)) } return frame, nil diff --git a/internal/uplink/tunnel_test.go b/internal/uplink/tunnel_test.go new file mode 100644 index 0000000..d248b78 --- /dev/null +++ b/internal/uplink/tunnel_test.go @@ -0,0 +1,205 @@ +package uplink + +import ( + "bytes" + "crypto/tls" + "crypto/x509" + "io" + "log/slog" + "sync" + "testing" + "time" + + "github.com/gorilla/websocket" + "github.com/srcfl/ftw-cloud/internal/store" + "github.com/srcfl/ftw/go/protocol" + "github.com/srcfl/ftw/go/protocol/cloudwire" +) + +// The service makes a connection the home's only after hello_ok has gone, +// so a phone that asks for one-second updates while a box connects can never +// put watch ahead of hello_ok. The box would close on it. +func TestNothingReachesABoxBeforeHelloOK(t *testing.T) { + s := newService(t) + b := newBox(t) + stop := make(chan struct{}) + var wg sync.WaitGroup + wg.Add(1) + go func() { + defer wg.Done() + for { + select { + case <-stop: + return + default: + s.up.Watch(b.home) + s.up.Tun(b.home, cloudwire.Tun{Dev: "x", Sid: 1, D: []byte{1}}) + } + } + }() + for range 20 { + // connect fails the test on anything but hello_ok after hello. + if _, err := b.connect(s, nil); err != nil { + t.Fatal(err) + } + _ = b.conn.Close() + } + close(stop) + wg.Wait() +} + +// After a restart the service holds every roster its boxes signed, and a +// home's phones are admitted before the box connects again. A roster that +// does not verify with the home's key is left out. +func TestRostersComeBackAfterARestart(t *testing.T) { + s := newService(t) + b := newBox(t) + if _, err := b.connect(s, nil); err != nil { + t.Fatal(err) + } + b.send(cloudwire.MsgRoster, b.roster(protocol.ScopeLiveRead)) + wait(t, "the roster is stored", func() bool { + h, ok, _ := s.reg.Get(b.home) + return ok && h.Roster != nil + }) + other := newBox(t) + if _, err := other.connect(s, nil); err != nil { + t.Fatal(err) + } + other.send(cloudwire.MsgRoster, other.roster(protocol.ScopeLiveRead)) + wait(t, "the second roster is stored", func() bool { + h, ok, _ := s.reg.Get(other.home) + return ok && h.Roster != nil + }) + // A roster in the file that its home's key did not sign. + forged, _ := cloudwire.Seal(b.key, other.home, cloudwire.MsgRoster, other.roster(protocol.ScopeLiveRead)) + raw, _ := forged.Encode() + if err := s.reg.SetRoster(other.home, 9, nil, raw, time.Now()); err != nil { + t.Fatal(err) + } + + fresh := store.New(nil) + up := New(fresh, s.reg, s.key, slog.New(slog.NewTextHandler(io.Discard, nil))) + n, err := up.Restore() + if err != nil || n != 1 { + t.Fatalf("restored %d: %v", n, err) + } + if _, err := fresh.For(store.Principal{Kind: store.Device, Home: b.home, DeviceKey: ownerKey}); err != nil { + t.Fatalf("a phone was refused after the restart: %v", err) + } + if _, err := fresh.For(store.Principal{Kind: store.Device, Home: other.home, DeviceKey: ownerKey}); err == nil { + t.Fatal("a roster its home did not sign admitted a phone") + } +} + +// A roster's grant has three shapes, and nothing else passes: a copy on with +// serve, a withdrawn copy with its id alone, or none at all. +func TestARosterGrantHasThreeShapes(t *testing.T) { + id := bytes.Repeat([]byte{5}, 16) + for name, c := range map[string]struct { + g cloudwire.Grant + ok bool + }{ + "on": {cloudwire.Grant{ID: id, Scopes: []string{protocol.ScopeLiveRead}, Purposes: []string{protocol.CloudPurposeServe}}, true}, + "withdrawn": {cloudwire.Grant{ID: id}, true}, + "off": {cloudwire.Grant{}, true}, + "scopes, no id": {cloudwire.Grant{Scopes: []string{protocol.ScopeLiveRead}}, false}, + "id and scopes": {cloudwire.Grant{ID: id, Scopes: []string{protocol.ScopeLiveRead}}, false}, + "a short id": {cloudwire.Grant{ID: id[:8], Purposes: []string{protocol.CloudPurposeServe}}, false}, + "a write scope": {cloudwire.Grant{ID: id, Scopes: []string{protocol.ScopeDispatchWrite}, Purposes: []string{protocol.CloudPurposeServe}}, false}, + } { + t.Run(name, func(t *testing.T) { + s := newService(t) + b := newBox(t) + if _, err := b.connect(s, nil); err != nil { + t.Fatal(err) + } + c.g.Controller = cloudwire.ControllerSourceful + r := b.roster() + r.Grant = c.g + b.send(cloudwire.MsgRoster, r) + if closed := b.closed(); closed == c.ok { + t.Fatalf("closed %v, want %v", closed, !c.ok) + } + }) + } +} + +// tunBox connects a box that takes tun and lists one phone that may open a +// tunnel, with the readable copy off. +func tunConnect(t *testing.T, s *service) *box { + t.Helper() + b := newBox(t) + pool := x509.NewCertPool() + pool.AddCert(s.srv.Certificate()) + d := websocket.Dialer{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS13}} + conn, _, err := d.Dial("wss://"+s.srv.Listener.Addr().String()+"/v1/box", nil) + if err != nil { + t.Fatal(err) + } + b.conn = conn + t.Cleanup(func() { _ = conn.Close() }) + cs := conn.NetConn().(*tls.Conn).ConnectionState() + ekm, _ := cs.ExportKeyingMaterial(cloudwire.EKMLabel, nil, 32) + var ch cloudwire.Challenge + _ = cloudwire.Unmarshal(b.read().B, &ch) + b.send(cloudwire.MsgHello, cloudwire.Hello{Header: b.header(), V: 1, PK: b.pk, Nonce: ch.Nonce, EKM: ekm, + BoxNonce: bytes.Repeat([]byte{4}, 32), Build: "test", Clock: cloudwire.ClockNTP, + Caps: []string{cloudwire.MsgRoster, cloudwire.CapTun}}) + if env := b.read(); env.T != cloudwire.MsgHelloOK { + t.Fatalf("got %q", env.T) + } + r := b.roster() + r.Grant = cloudwire.Grant{Controller: cloudwire.ControllerSourceful} + r.Devices[0].Tun = true + b.send(cloudwire.MsgRoster, r) + return b +} + +func (b *box) tun(t cloudwire.Tun) { + body, _ := cloudwire.Marshal(t) + raw, _ := cloudwire.Envelope{T: cloudwire.MsgTun, B: body}.Encode() + frame, _ := cloudwire.EncodeFrame(raw) + _ = b.conn.WriteMessage(websocket.BinaryMessage, frame) +} + +// tun from the box is not held to its message rate, and a tun for a phone +// that is not there ends the box's session at once. +func TestATunForNoPhoneIsEndedAndTunIsNotTheBoxsRate(t *testing.T) { + s := newService(t) + b := tunConnect(t, s) + wait(t, "the roster", func() bool { + _, err := s.store.For(store.Principal{Kind: store.Device, Home: b.home, DeviceKey: ownerKey}) + return err == nil + }) + for i := range rateBurst + 20 { + b.tun(cloudwire.Tun{Dev: "owner", Sid: uint32(i), D: []byte{1}}) + } + ends := 0 + for ends < rateBurst+20 { + env := b.read() + var t2 cloudwire.Tun + if env.T != cloudwire.MsgTun || cloudwire.Unmarshal(env.B, &t2) != nil || !t2.Ended() { + t.Fatalf("got %q %+v", env.T, t2) + } + ends++ + } +} + +// Past a home's tun limits the service drops and counts, and the box stays +// connected. +func TestPastTheTunLimitsTheServiceDrops(t *testing.T) { + s := newService(t) + b := tunConnect(t, s) + wait(t, "the box is the home's", func() bool { return s.up.Tun(b.home, cloudwire.Tun{Dev: "owner", Sid: 1}) }) + big := make([]byte, cloudwire.TunDataMax) + for range 300 { + s.up.Tun(b.home, cloudwire.Tun{Dev: "owner", Sid: 1, D: big}) + } + if s.up.TunDropped.Load() == 0 { + t.Fatal("nothing was dropped past 4 MiB in a burst") + } + if !s.up.Tun(b.home, cloudwire.Tun{Dev: "owner", Sid: 1}) { + t.Fatal("the box is gone") + } +} diff --git a/internal/uplink/uplink.go b/internal/uplink/uplink.go index 49bb46f..a93c3c1 100644 --- a/internal/uplink/uplink.go +++ b/internal/uplink/uplink.go @@ -1,11 +1,13 @@ -// Package uplink serves /v1/box, where each box that has turned FTW Cloud on -// keeps one connection. docs/protocol-cloud.md in srcfl/ftw is the wire -// format; go/protocol/cloudwire holds its bytes. +// Package uplink serves /v1/box, where each linked box keeps one +// connection. docs/protocol-cloud.md in srcfl/ftw is the wire format; +// go/protocol/cloudwire holds its bytes. // // The service learns a home only from its box: the home id is a hash of the // key that signed hello on this TLS session, never a field anyone sends, and -// every message after it must carry that key's signature. What the service -// may say back is the closed set in downlink.go. +// every message after it but tun must carry that key's signature. tun is a +// phone's own Noise session with the box, which the service routes by device +// and never reads. What the service may say back is the closed set in +// downlink.go. package uplink import ( @@ -43,9 +45,21 @@ const ( // writeTimeout: a write blocked this long is a dead socket. writeTimeout = 10 * time.Second // A box sends a few messages a second at most. Past the burst at this - // rate the connection closes; one home cannot occupy the service. + // rate the connection closes; one home cannot occupy the service. tun + // is counted apart. rateBurst = 60 ratePerSec = 10 + // tun per home and direction, as the relay allows one stream. Past it + // the service drops tun and counts the drop; Noise transport survives a + // lost message. + tunMsgBurst = 240 + tunMsgPerSec = 60 + tunByteBurst = 4 << 20 + tunBytePerSec = 512 << 10 + // maxPending bounds the pairing codes one roster may list, and + // maxPendingTTL how long one may last. + maxPending = 4 + maxPendingTTL = 3600 ) // Uplink is the /v1/box handler. @@ -62,6 +76,8 @@ type Uplink struct { // Dropped counts box messages of a type this service does not know. A // newer box works with an older service by this rule. Dropped atomic.Int64 + // TunDropped counts tun messages past a home's limits. + TunDropped atomic.Int64 // OnConnect runs after a box's hello_ok, so the app sessions can ask a // box that just came back for one-second updates and push can stop @@ -112,7 +128,10 @@ func (u *Uplink) ServeHTTP(w http.ResponseWriter, r *http.Request) { } defer conn.Close() conn.SetReadLimit(maxBoxFrame) - c := &boxConn{u: u, conn: conn, ekm: ekm, tokens: rateBurst, refilled: u.now()} + now := u.now() + c := &boxConn{u: u, conn: conn, ekm: ekm, tokens: rateBurst, refilled: now, + upMsgs: newBucket(tunMsgBurst, tunMsgPerSec, now), upBytes: newBucket(tunByteBurst, tunBytePerSec, now), + downMsgs: newBucket(tunMsgBurst, tunMsgPerSec, now), downBytes: newBucket(tunByteBurst, tunBytePerSec, now)} c.write = c.writeFrame if err := c.run(); err != nil && !websocket.IsCloseError(err, websocket.CloseNormalClosure, websocket.CloseGoingAway) { u.log.Debug("box uplink closed", "err", err) @@ -130,6 +149,58 @@ func (u *Uplink) Watch(home string) { } } +// Tun sends a phone's tun to its home's box. t.Dev is the device the app +// session's own handshake matched, never one a phone named. False means no +// box takes it now: none is connected, or it does not take tun. A message +// past the home's limits is dropped and counted, and Tun still says true, +// because the box is there. +func (u *Uplink) Tun(home string, t cloudwire.Tun) bool { + u.mu.Lock() + c := u.conns[home] + u.mu.Unlock() + if c == nil || !c.tun { + return false + } + c.downMu.Lock() + ok := tunAllowed(&c.downMsgs, &c.downBytes, len(t.D), u.now()) + c.downMu.Unlock() + if !ok { + u.TunDropped.Add(1) + return true + } + return c.send(tun(t)) == nil +} + +// Restore holds each home's last signed roster again after a restart, read +// back from the registry, so its phones are admitted before its box has +// reconnected. A roster that does not verify with the home's key is left +// out. The roster's time to live counts from when it first arrived. +func (u *Uplink) Restore() (int, error) { + homes, err := u.registry.Rosters() + if err != nil { + return 0, err + } + n := 0 + for _, h := range homes { + pub, err := cloudwire.ParsePublicKey(h.PK) + if err != nil || cloudwire.HomeID(h.PK) != h.ID { + continue + } + env, err := cloudwire.DecodeEnvelope(h.Roster) + if err != nil || env.T != cloudwire.MsgRoster || cloudwire.Verify(pub, h.ID, env) != nil { + u.log.Warn("a stored roster does not verify; its home waits for its box") + continue + } + var r cloudwire.Roster + if cloudwire.Unmarshal(env.B, &r) != nil { + continue + } + u.store.Seed(h.ID, &r, h.RosterAt) + n++ + } + return n, nil +} + // boxConn is one box's connection. type boxConn struct { u *Uplink @@ -143,11 +214,53 @@ type boxConn struct { pub *ecdsa.PublicKey handle *store.Handle purged bool + // tun is whether the box listed tun in hello.caps. An older box closes + // the uplink on a type it does not know, so it is sent none. + tun bool // bye is the reason the box gave for leaving, if it gave one. bye string tokens float64 refilled time.Time + + // The tunnel's limits: up from the box, read on its goroutine; down + // from the home's phones, from theirs. + upMsgs, upBytes bucket + downMu sync.Mutex + downMsgs bucket + downBytes bucket +} + +// bucket is a token bucket. +type bucket struct { + tokens, burst, rate float64 + at time.Time +} + +func newBucket(burst, rate float64, now time.Time) bucket { + return bucket{tokens: burst, burst: burst, rate: rate, at: now} +} + +func (b *bucket) take(n float64, now time.Time) bool { + b.tokens = min(b.burst, b.tokens+now.Sub(b.at).Seconds()*b.rate) + b.at = now + if b.tokens < n { + return false + } + b.tokens -= n + return true +} + +// tunAllowed takes one message of n bytes from a direction's two buckets. +func tunAllowed(msgs, bytes *bucket, n int, now time.Time) bool { + if !msgs.take(1, now) { + return false + } + if !bytes.take(float64(n), now) { + msgs.tokens++ + return false + } + return true } func (c *boxConn) writeFrame(frame []byte) error { @@ -196,6 +309,14 @@ func (c *boxConn) run() error { if c.handle, err = c.u.store.For(store.Principal{Kind: store.Box, Home: c.home}); err != nil { return err } + } + if err := c.send(helloOK{BoxNonce: hello.BoxNonce, EKM: c.ekm, Have: map[string]string{}, RosterSeq: c.rosterSeq()}); err != nil { + return err + } + // The connection becomes the home's only after hello_ok has gone, so a + // watch or a tun from a phone can never reach the box before it: the + // box closes on anything else while it waits for hello_ok. + if !c.purged { gen := c.handle.Connected() defer c.handle.Disconnected(gen) c.u.mu.Lock() @@ -216,13 +337,9 @@ func (c *boxConn) run() error { c.u.OnGone(c.home, c.bye) } }() - } - - if err := c.send(helloOK{BoxNonce: hello.BoxNonce, EKM: c.ekm, Have: map[string]string{}, RosterSeq: c.rosterSeq()}); err != nil { - return err - } - if !c.purged && c.u.OnConnect != nil { - go c.u.OnConnect(c.home) + if c.u.OnConnect != nil { + go c.u.OnConnect(c.home) + } } done := make(chan struct{}) @@ -238,13 +355,19 @@ func (c *boxConn) run() error { return err } _ = c.conn.SetReadDeadline(time.Now().Add(readTimeout)) - if !c.allow() { - return fmt.Errorf("%w: faster than %d messages a second", errProtocol, ratePerSec) - } env, err := parseUplink(kind, msg) if err != nil { return err } + if env.T == cloudwire.MsgTun && !c.purged { + if err := c.fromBox(env); err != nil { + return err + } + continue + } + if !c.allow() { + return fmt.Errorf("%w: faster than %d messages a second", errProtocol, ratePerSec) + } stop, err := c.apply(env) if err != nil || stop { return err @@ -269,10 +392,28 @@ func (c *boxConn) hello(env cloudwire.Envelope, nonce []byte) (cloudwire.Hello, if !bytes.Equal(h.Nonce, nonce) || !bytes.Equal(h.EKM, c.ekm) || len(h.BoxNonce) != cloudwire.NonceBytes { return h, fmt.Errorf("%w: hello is not bound to this challenge and TLS session", errProtocol) } - c.home, c.pub = home, pub + c.home, c.pub, c.tun = home, pub, slices.Contains(h.Caps, cloudwire.CapTun) return h, nil } +// fromBox routes one tun from the box to the phone whose session it is. It +// is not signed and never read: d is the phone's and the box's. A phone that +// has gone gets its session ended at the box. +func (c *boxConn) fromBox(env cloudwire.Envelope) error { + var t cloudwire.Tun + if len(env.Sig) != 0 || cloudwire.Unmarshal(env.B, &t) != nil || len(t.D) > cloudwire.TunDataMax || t.Dev == "" { + return fmt.Errorf("%w: a tun the service cannot route", errProtocol) + } + if !tunAllowed(&c.upMsgs, &c.upBytes, len(t.D), c.u.now()) { + c.u.TunDropped.Add(1) + return nil + } + if !c.handle.Tun(t) && !t.Ended() && c.tun { + return c.send(tun{Dev: t.Dev, Sid: t.Sid}) + } + return nil +} + func (c *boxConn) rosterSeq() uint64 { var seq uint64 if c.handle != nil { @@ -362,6 +503,10 @@ func (c *boxConn) apply(env cloudwire.Envelope) (stop bool, err error) { c.u.Dropped.Add(1) return false, nil } + if !c.handle.Serving() { + // The readable copy is off: the service holds no readings. + return false, nil + } raw, err := env.Encode() if err != nil { return true, err @@ -398,8 +543,14 @@ func (c *boxConn) roster(env cloudwire.Envelope) error { } return true } - if len(r.Grant.ID) != cloudwire.GrantIDBytes || r.Grant.Controller == "" || r.TTLS <= 0 || - !within(r.Grant.Scopes, protocol.CloudScopes) || !within(r.Grant.Purposes, protocol.CloudPurposes) { + // A grant has three shapes: a copy with serve, a withdrawn one with its + // id alone, or none at all. + g := r.Grant + on := len(g.ID) == cloudwire.GrantIDBytes && slices.Contains(g.Purposes, protocol.CloudPurposeServe) + bare := len(g.Scopes) == 0 && len(g.Purposes) == 0 + withdrawn := len(g.ID) == cloudwire.GrantIDBytes && bare + if !(on || withdrawn || (len(g.ID) == 0 && bare)) || g.Controller == "" || r.TTLS <= 0 || + !within(g.Scopes, protocol.CloudScopes) || !within(g.Purposes, protocol.CloudPurposes) { return fmt.Errorf("%w: a roster outside what FTW Cloud may hold", errProtocol) } for _, d := range r.Devices { @@ -407,9 +558,22 @@ func (c *boxConn) roster(env cloudwire.Envelope) error { return fmt.Errorf("%w: a roster device the service cannot use", errProtocol) } } + if len(r.Pending) > maxPending { + return fmt.Errorf("%w: %d pairing codes in one roster", errProtocol, len(r.Pending)) + } + for _, p := range r.Pending { + if len(p.Hash) != cloudwire.PairingHashBytes || p.TTLS <= 0 || p.TTLS > maxPendingTTL { + return fmt.Errorf("%w: a pairing code the service cannot hold", errProtocol) + } + } if !c.handle.SetRoster(&r) { return nil // older than what is held: ignored } + if withdrawn { + if err := c.dropCopy(g.ID); err != nil { + return err + } + } if c.u.OnRoster != nil { c.u.OnRoster(c.home, &r) } @@ -429,6 +593,19 @@ func (c *boxConn) roster(env cloudwire.Envelope) error { return c.u.registry.PruneSuspensions(c.home, epochs) } +// dropCopy deletes what the service holds under a grant its owner withdrew, +// the readings and the extension tokens that read them, and sends the signed +// receipt. The home, its roster and its tunnel stay. A roster that names the +// same withdrawn grant again gets the receipt again. +func (c *boxConn) dropCopy(grantID []byte) error { + c.handle.DropCopy() + if err := c.u.registry.TokenPrune(c.home, func(string, uint64) bool { return false }); err != nil { + return err + } + c.u.log.Info("a home's readable copy deleted at its box's request") + return c.send(purged{AtMs: c.u.now().UnixMilli(), What: cloudwire.PurgedCopy, GrantID: grantID}) +} + // purge deletes the home and sends the signed receipt. For a home already // purged, the receipt goes again. func (c *boxConn) purge() error { diff --git a/internal/uplink/uplink_test.go b/internal/uplink/uplink_test.go index bc6c62d..eaa9a2c 100644 --- a/internal/uplink/uplink_test.go +++ b/internal/uplink/uplink_test.go @@ -327,7 +327,8 @@ func TestASecondUplinkReplacesTheFirst(t *testing.T) { func TestDownlinkIsTheProtocolsClosedSet(t *testing.T) { s := newService(t) all := []down{challenge{V: 1, Nonce: make([]byte, 32)}, helloOK{Have: map[string]string{}}, - watch{Hz: 1}, resync{What: []string{cloudwire.ResyncFull}}, purged{What: cloudwire.PurgedHome}} + watch{Hz: 1}, resync{What: []string{cloudwire.ResyncFull}}, purged{What: cloudwire.PurgedHome}, + tun{Dev: "q5Lw0Zb1", Sid: 1, D: []byte{1}}} var names []string for _, m := range all { frame, err := s.up.encodeDown("home", m) @@ -380,3 +381,29 @@ func FuzzBoxMessage(f *testing.F) { _, _ = c.apply(env) }) } + +// Whatever tun body a box sends, the service routes it, ends it or closes, +// and never panics. +func FuzzBoxTun(f *testing.F) { + key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + pk, _ := cloudwire.PublicKeyBytesOf(&key.PublicKey) + home := cloudwire.HomeID(pk) + for _, seed := range []any{ + cloudwire.Tun{Dev: "owner", Sid: 1, D: []byte{1}}, + cloudwire.Tun{Dev: "owner", Sid: 1}, + map[string]any{"dev": 7, "sid": "x", "d": "y"}, + } { + body, _ := cloudwire.Marshal(seed) + f.Add(body) + } + f.Fuzz(func(t *testing.T, body []byte) { + st := store.New(nil) + u := New(st, nil, key, slog.New(slog.NewTextHandler(io.Discard, nil))) + handle, _ := st.For(store.Principal{Kind: store.Box, Home: home}) + now := time.Now() + c := &boxConn{u: u, home: home, pub: &key.PublicKey, handle: handle, tun: true, + write: func([]byte) error { return nil }, + upMsgs: newBucket(tunMsgBurst, tunMsgPerSec, now), upBytes: newBucket(tunByteBurst, tunBytePerSec, now)} + _ = c.fromBox(cloudwire.Envelope{T: cloudwire.MsgTun, B: body}) + }) +} From 462813deb948c411431df12fa7bdf463d67dc04b Mon Sep 17 00:00:00 2001 From: Fredrik Ahlgren Date: Thu, 15 Oct 2026 17:28:15 +0200 Subject: [PATCH 2/4] fix: review of the tunnel (B, A, D, C and E-N, K, L) - B: a box that dials again ends the tunnels its older uplink carried, so phones hear it even while the old socket is still open. - A: a box sending tun at twice the home's limits is closed; under that the service drops and counts. The limits are now cloudwire's, shared with the box, which paces its bulk under them (srcfl/ftw#1635). - D: readings go from memory whenever a roster does not serve. - E, F, M: a restart does not restore pairing codes; a key the roster lists never enters by pairing; the hash is spent only when the session opens. - G: a roster that takes a device's tunnel away ends its session. - H, N: the box hears the end of a phone's tunnel however the app session ends; a d larger than any session message closes the phone with 4400. - I: push hooks carry the uplink's connection number, so the late end of an older connection never starts box.unreachable for a box that is back. - K: suspensions are read from memory, so admission under a home's lock never waits for SQLite. - L: dropped tun is logged once a minute as a count for the whole service. - C: the guide and README say what holds now, and list each part's size; the budgets are Fredrik's call. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Fredrik Ahlgren --- CLAUDE.md | 29 +++++- README.md | 10 +- cmd/ftwcloudd/main.go | 4 +- go.mod | 2 +- go.sum | 4 +- internal/appsession/appsession.go | 23 +++-- internal/push/push.go | 44 +++++++-- internal/push/push_test.go | 38 +++++--- internal/push/resume_test.go | 30 ++++-- internal/registry/registry.go | 79 +++++++++++++--- internal/server/e2e_test.go | 4 +- internal/server/server.go | 7 ++ internal/server/tunnel_e2e_test.go | 65 ++++++++++++- internal/store/session.go | 13 ++- internal/store/store.go | 48 ++++++++-- internal/store/tunnel_test.go | 142 +++++++++++++++++++++++++++++ internal/uplink/hooks_test.go | 4 +- internal/uplink/tunnel_test.go | 13 +++ internal/uplink/uplink.go | 42 ++++++--- internal/uplink/uplink_test.go | 3 +- 20 files changed, 507 insertions(+), 97 deletions(-) create mode 100644 internal/store/tunnel_test.go diff --git a/CLAUDE.md b/CLAUDE.md index e784d9f..4ad9eeb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,9 +19,13 @@ maintains it. names. - **One way to a home:** `store.For(principal)`. No listing, no second lookup, no principal for staff. A new route needs a row in `server.Routes` and must - pass `TestNoRouteReadsAHome`. + pass `TestNoRouteReadsAHome`. The one other way in is `store.Seed`, called + once at start: it puts back the rosters the registry holds, each checked + against its home's key, and reads nothing out. - **The home id comes from the key that signed hello** on this connection, - never from a body field. Every later box message must verify with that key. + never from a body field. Every later box message but `tun` must verify + with that key. `tun` is unsigned: its `d` is a phone's ciphertext, which + the service routes by `dev` and `sid` and never reads. - **No readings on disk.** Memory holds readings, and only while the roster's grant serves a copy; SQLite holds only the home id, the box key, the signed roster, owners' suspensions, tombstones, APNs registrations, the hashes of @@ -46,9 +50,24 @@ maintains it. - Check open PRs on the files you touch; an open PR has right of way. - A wire change is a pair: `srcfl/ftw` (`go/protocol` and its golden vectors) first, then this repository, moving the pin in `go.mod`. Say so in both PRs. -- Keep the service small. The budget is about 3,000 lines of Go; tiles go - first if it is overrun. It is over today; the one-door plan gives each part - a budget of its own. +- Keep the service small. The old budget was about 3,000 lines of Go in all, + and the service is over it. The one-door plan gives each part a budget of + its own, and the numbers are Fredrik's call. Until he sets them, a PR that + grows a part says by how much. Lines of Go without tests, at + srcfl/ftw-cloud#15: + + | Part | Lines | + |---|---| + | `internal/appsession` (app session, one-shot read) | 1,020 | + | `internal/store` | 807 | + | `internal/uplink` | 719 | + | `internal/registry` | 694 | + | `cmd/ftwcloudd` | 553 | + | `internal/push` | 542 | + | `internal/apns` | 299 | + | `internal/server` | 205 | + | `internal/token` | 170 | + | All | 5,009 | - Nothing is deployed from here without the owner. `deploy/` describes the running host; it does not run itself. - Keep task breakdowns and agent notes out of the repository. diff --git a/README.md b/README.md index b4b4715..3c66572 100644 --- a/README.md +++ b/README.md @@ -41,9 +41,13 @@ to it yet. way. The box session's first message may ride in the app session's message 1 (`now.tun`), so it is up one round trip after TLS. An empty `d` ends a session: the box hears when the phone's app session ends, and the phone - when the box's uplink does. tun has its own queues and limits per home and - direction: 240 messages then 60 a second, 4 MiB then 512 KiB a second. A - roster device needs `tun` or a readable copy to be admitted at all. + when the box's uplink does, or when the box dials again. From the box, + tun rides a queue of its own per phone session, apart from the readings; + from a phone it is written straight to the box's socket. Each direction + has limits per home: 240 messages then 60 a second, 4 MiB then 512 KiB a + second. Past them the service drops and counts, and logs the count each + minute; a box at twice them is closed, and so is a phone. A roster device + needs `tun` or a readable copy to be admitted at all. - **Pairing through the service.** While a QR code is on the box's screen, its roster lists the code's hash. A new phone proves the hash in `now.pair` and is admitted once, as a device that may only open a tunnel; the box diff --git a/cmd/ftwcloudd/main.go b/cmd/ftwcloudd/main.go index 2da6f0f..2bc334f 100644 --- a/cmd/ftwcloudd/main.go +++ b/cmd/ftwcloudd/main.go @@ -145,11 +145,11 @@ func serve(args []string) error { srv.App.Tunnel = up } app := srv.App - up.OnConnect = func(home string) { + up.OnConnect = func(home string, conn uint64) { if app != nil { app.BoxConnected(home) } - pushSvc.BoxConnected(home) + pushSvc.BoxConnected(home, conn) } // TLS ends here and nowhere else: each box binds its handshake to this diff --git a/go.mod b/go.mod index 48c660f..17a2003 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.26.0 require ( github.com/fxamacker/cbor/v2 v2.9.4 github.com/gorilla/websocket v1.5.3 - github.com/srcfl/ftw/go/protocol v0.0.0-20261015144614-0a5e04691065 + github.com/srcfl/ftw/go/protocol v0.0.0-20261015152627-0970e7b36690 golang.org/x/crypto v0.57.0 golang.org/x/sys v0.48.0 modernc.org/sqlite v1.59.0 diff --git a/go.sum b/go.sum index 17c3405..4439303 100644 --- a/go.sum +++ b/go.sum @@ -16,8 +16,8 @@ github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOF github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= -github.com/srcfl/ftw/go/protocol v0.0.0-20261015144614-0a5e04691065 h1:tKaqxGV1K69ljIBH1szkDz3zxqeZKdu5eanSeDY9qNo= -github.com/srcfl/ftw/go/protocol v0.0.0-20261015144614-0a5e04691065/go.mod h1:VEjr/o0yZijJ1nApuJK+nak6FxJ/fubAUBNOunqNrUU= +github.com/srcfl/ftw/go/protocol v0.0.0-20261015152627-0970e7b36690 h1:wnkUzAoSXPBuosc2awje2vZdUQdTwiXVYWwfOZZ5P7w= +github.com/srcfl/ftw/go/protocol v0.0.0-20261015152627-0970e7b36690/go.mod h1:VEjr/o0yZijJ1nApuJK+nak6FxJ/fubAUBNOunqNrUU= github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M= diff --git a/internal/appsession/appsession.go b/internal/appsession/appsession.go index e0e8d08..9713d4d 100644 --- a/internal/appsession/appsession.go +++ b/internal/appsession/appsession.go @@ -235,6 +235,8 @@ func (s *Server) serve(conn *websocket.Conn, home string) (int, error) { c := &phoneConn{s: s, conn: conn, dev: dev, home: home, tokens: rateBurst, refilled: s.now(), tunTokens: tunBurst, tunAt: s.now(), sess: sess, replies: make(chan []byte, 1), quit: make(chan struct{})} + // Whatever ends this session from here on, the box hears its tunnel end. + defer c.endTun() // The box session's message 1, when it rode in message 1, goes to the // box before message 2 goes back: the box's answer then follows message // 2 by one leg, and the session is up one round trip after TLS. @@ -257,7 +259,6 @@ func (s *Server) serve(conn *websocket.Conn, home string) (int, error) { c.setLooking(hz >= cloudwire.HzLooking) defer c.stopLooking() - defer c.endTun() return c.run(sess) } @@ -517,11 +518,15 @@ func (c *phoneConn) readLoop() (int, error) { // tun forwards the phone's own session to its box, under the device id the // handshake proved. A device without tun in the roster sends nothing. When -// no box takes it, the phone hears its session ended. -func (c *phoneConn) tun(t cloudwire.AppTun) error { +// no box takes it, the phone hears its session ended. A d larger than any +// session message breaks the protocol. +func (c *phoneConn) tun(t cloudwire.AppTun) (int, error) { + if len(t.D) > cloudwire.TunDataMax { + return cloudwire.CloseProtocol, errProtocol + } dev := c.dev.Device() - if !dev.Tun || len(t.D) > cloudwire.TunDataMax { - return nil + if !dev.Tun { + return 0, nil } c.tunMu.Lock() c.tunSid, c.hasTun = t.Sid, len(t.D) > 0 @@ -530,12 +535,12 @@ func (c *phoneConn) tun(t cloudwire.AppTun) error { c.sess.SetSid(t.Sid) } if c.s.Tunnel != nil && c.s.Tunnel.Tun(c.home, cloudwire.Tun{Dev: dev.ID, Sid: t.Sid, D: t.D}) { - return nil + return 0, nil } if len(t.D) == 0 { - return nil + return 0, nil } - return c.say(cloudwire.MsgTun, cloudwire.AppTun{Sid: t.Sid}) + return 0, c.say(cloudwire.MsgTun, cloudwire.AppTun{Sid: t.Sid}) } // tunEarly forwards the box session's message 1 that rode in message 1. No @@ -591,7 +596,7 @@ func (c *phoneConn) apply(frame []byte) (int, error) { if err := cloudwire.Unmarshal(env.B, &t); err != nil { return cloudwire.CloseProtocol, err } - return 0, c.tun(t) + return c.tun(t) } // tun has limits of its own, per home; everything else counts here. if len(frame) > maxPlainMessage || !c.allow() { diff --git a/internal/push/push.go b/internal/push/push.go index 7a643b0..1d1b9af 100644 --- a/internal/push/push.go +++ b/internal/push/push.go @@ -126,8 +126,11 @@ type Service struct { type homeState struct { online bool // gen moves on every connect and disconnect, so a timer set for one - // outage cannot fire into the next. + // outage cannot fire into the next. conn is the newest uplink + // connection heard of, so the end of an older one, heard late, does + // not start the switch for a box that is back. gen uint64 + conn uint64 stop func() bool told bool sent map[string][]time.Time @@ -315,9 +318,16 @@ func pricedUntil(p cloudwire.Prices) int64 { } // BoxConnected stops the dead man's switch for a home whose box is back. -func (s *Service) BoxConnected(home string) { +// conn is the uplink's connection number for the home, which rises with +// every connection. +func (s *Service) BoxConnected(home string, conn uint64) { s.mu.Lock() st := s.state(home) + if conn < st.conn { + s.mu.Unlock() + return + } + st.conn = conn st.gen++ st.online, st.told = true, false if st.stop != nil { @@ -333,8 +343,15 @@ func (s *Service) BoxConnected(home string) { // BoxGone starts the dead man's switch. why is the box's bye reason, empty // when the socket just dropped. A bye for an update holds box.unreachable // back until the box has been away for cloudwire.UpdateGraceMs; a revoke -// ends it, since the home is gone. -func (s *Service) BoxGone(home, why string) { +// ends it, since the home is gone. The end of a connection older than one +// already back is ignored. +func (s *Service) BoxGone(home, why string, conn uint64) { + s.mu.Lock() + stale := s.homes[home] != nil && conn < s.homes[home].conn + s.mu.Unlock() + if stale { + return + } if why == cloudwire.ByeRevoked { s.mu.Lock() if st := s.homes[home]; st != nil && st.stop != nil { @@ -352,20 +369,26 @@ func (s *Service) BoxGone(home, why string) { wait = time.Duration(cloudwire.UpdateGraceMs) * time.Millisecond } s.log.Info("a box went away", "bye", why, "out_of_reach_after", wait) + if !s.arm(home, wait, conn) { + return + } // Kept on disk, so a restart in the meantime neither forgets the alert // nor sends it twice. if err := s.reg.AwaySet(home, s.now().Add(wait)); err != nil { s.log.Warn("could not record a box's absence", "err", err) } - s.arm(home, wait) } -// arm starts the wait for box.unreachable. The home's generation moves, so -// a timer from an earlier absence cannot fire into this one. -func (s *Service) arm(home string, wait time.Duration) { +// arm starts the wait for box.unreachable, unless a newer connection than +// conn is already up. The home's generation moves, so a timer from an +// earlier absence cannot fire into this one. +func (s *Service) arm(home string, wait time.Duration, conn uint64) bool { s.mu.Lock() defer s.mu.Unlock() st := s.state(home) + if conn < st.conn { + return false + } st.gen++ st.online, st.told = false, false if st.stop != nil { @@ -373,6 +396,7 @@ func (s *Service) arm(home string, wait time.Duration) { } gen := st.gen st.stop = s.after(wait, func() { s.unreachable(home, gen) }) + return true } // Resume starts the dead man's switch again after a restart. A home recorded @@ -395,7 +419,7 @@ func (s *Service) Resume() error { s.mu.Unlock() continue } - s.arm(a.Home, max(0, a.Due.Sub(now))) + s.arm(a.Home, max(0, a.Due.Sub(now)), 0) } homes, err := s.reg.PushHomes(cloudwire.PushKindAlert) if err != nil { @@ -409,7 +433,7 @@ func (s *Service) Resume() error { if err := s.reg.AwaySet(home, now.Add(wait)); err != nil { return err } - s.arm(home, wait) + s.arm(home, wait, 0) } return nil } diff --git a/internal/push/push_test.go b/internal/push/push_test.go index 787ceab..8546020 100644 --- a/internal/push/push_test.go +++ b/internal/push/push_test.go @@ -94,6 +94,7 @@ type rig struct { apple *fakeApple clock *timers home string + conn uint64 owner cloudwire.Device viewer cloudwire.Device all []string @@ -120,6 +121,15 @@ func newRig(t *testing.T) *rig { return r } +// connected and gone are the uplink's hooks for one connection after +// another. +func (r *rig) connected() { + r.conn++ + r.svc.BoxConnected(r.home, r.conn) +} + +func (r *rig) gone(why string) { r.svc.BoxGone(r.home, why, r.conn) } + // drain runs what the uplink queued, as Run would. func (r *rig) drain() { for { @@ -222,7 +232,7 @@ func TestASuspendedDeviceGetsNothingUntilTheSuspensionEnds(t *testing.T) { t.Fatalf("while suspended, sent to %v", got) } // box.unreachable takes the same road. - r.svc.BoxGone(r.home, "") + r.gone("") r.clock.advance(10 * time.Minute) if got := tokens(); len(got) != 1 || got[0] != "owner-token" { t.Fatalf("unreachable went to %v", got) @@ -301,8 +311,8 @@ func unreachableSent(t *testing.T, r *rig) int { func TestBoxUnreachableAfterTenMinutesOncePerOutage(t *testing.T) { r := newRig(t) _ = r.register(t, r.owner, r.all, "tok") - r.svc.BoxConnected(r.home) - r.svc.BoxGone(r.home, "") + r.connected() + r.gone("") r.clock.advance(599 * time.Second) if n := unreachableSent(t, r); n != 0 { t.Fatal("before ten minutes") @@ -316,8 +326,8 @@ func TestBoxUnreachableAfterTenMinutesOncePerOutage(t *testing.T) { t.Fatalf("sent %d over one outage", n) } // Back and away again is a new outage. - r.svc.BoxConnected(r.home) - r.svc.BoxGone(r.home, cloudwire.ByeShutdown) + r.connected() + r.gone(cloudwire.ByeShutdown) r.clock.advance(601 * time.Second) if n := unreachableSent(t, r); n != 2 { t.Fatalf("sent %d over two outages", n) @@ -327,10 +337,10 @@ func TestBoxUnreachableAfterTenMinutesOncePerOutage(t *testing.T) { func TestABoxBackInTimeSaysNothing(t *testing.T) { r := newRig(t) _ = r.register(t, r.owner, r.all, "tok") - r.svc.BoxConnected(r.home) - r.svc.BoxGone(r.home, "") + r.connected() + r.gone("") r.clock.advance(5 * time.Minute) - r.svc.BoxConnected(r.home) + r.connected() r.clock.advance(time.Hour) if n := unreachableSent(t, r); n != 0 { t.Fatalf("sent %d for a box that came back", n) @@ -342,19 +352,19 @@ func TestABoxBackInTimeSaysNothing(t *testing.T) { func TestAPlannedUpdateHoldsItBackForAnHour(t *testing.T) { r := newRig(t) _ = r.register(t, r.owner, r.all, "tok") - r.svc.BoxConnected(r.home) - r.svc.BoxGone(r.home, cloudwire.ByeUpdate) + r.connected() + r.gone(cloudwire.ByeUpdate) r.clock.advance(20 * time.Minute) if n := unreachableSent(t, r); n != 0 { t.Fatal("an update was reported as out of reach") } - r.svc.BoxConnected(r.home) + r.connected() r.clock.advance(2 * time.Hour) if n := unreachableSent(t, r); n != 0 { t.Fatal("an update that came back was reported") } - r.svc.BoxGone(r.home, cloudwire.ByeUpdate) + r.gone(cloudwire.ByeUpdate) r.clock.advance(time.Hour + time.Second) if n := unreachableSent(t, r); n != 1 { t.Fatalf("an update still away after an hour: %d", n) @@ -364,8 +374,8 @@ func TestAPlannedUpdateHoldsItBackForAnHour(t *testing.T) { func TestARevokedHomeHasNoDeadMan(t *testing.T) { r := newRig(t) _ = r.register(t, r.owner, r.all, "tok") - r.svc.BoxConnected(r.home) - r.svc.BoxGone(r.home, cloudwire.ByeRevoked) + r.connected() + r.gone(cloudwire.ByeRevoked) r.clock.advance(2 * time.Hour) if n := unreachableSent(t, r); n != 0 { t.Fatal("a revoked home was reported out of reach") diff --git a/internal/push/resume_test.go b/internal/push/resume_test.go index a34dfbc..eea31f9 100644 --- a/internal/push/resume_test.go +++ b/internal/push/resume_test.go @@ -21,8 +21,8 @@ func (r *rig) restart(t *testing.T) { func TestARestartKeepsTheWaitForBoxUnreachable(t *testing.T) { r := newRig(t) _ = r.register(t, r.owner, r.all, "tok") - r.svc.BoxConnected(r.home) - r.svc.BoxGone(r.home, "") + r.connected() + r.gone("") r.clock.advance(400 * time.Second) r.restart(t) r.clock.advance(199 * time.Second) @@ -39,8 +39,8 @@ func TestARestartKeepsTheWaitForBoxUnreachable(t *testing.T) { func TestARestartDoesNotRepeatBoxUnreachable(t *testing.T) { r := newRig(t) _ = r.register(t, r.owner, r.all, "tok") - r.svc.BoxConnected(r.home) - r.svc.BoxGone(r.home, "") + r.connected() + r.gone("") r.clock.advance(601 * time.Second) if n := unreachableSent(t, r); n != 1 { t.Fatalf("sent %d", n) @@ -57,10 +57,10 @@ func TestARestartDoesNotRepeatBoxUnreachable(t *testing.T) { func TestARestartGivesAConnectedBoxItsWholeWait(t *testing.T) { r := newRig(t) _ = r.register(t, r.owner, r.all, "tok") - r.svc.BoxConnected(r.home) + r.connected() r.restart(t) r.clock.advance(30 * time.Second) - r.svc.BoxConnected(r.home) + r.connected() r.clock.advance(time.Hour) if n := unreachableSent(t, r); n != 0 { t.Fatalf("sent %d for a box that came back", n) @@ -72,3 +72,21 @@ func TestARestartGivesAConnectedBoxItsWholeWait(t *testing.T) { t.Fatalf("sent %d for a box that never came back after a restart", n) } } + +// The end of an older connection, heard after the next one is up, does not +// start the switch for a box that is back. +func TestALateEndOfAnOlderConnectionSaysNothing(t *testing.T) { + r := newRig(t) + _ = r.register(t, r.owner, r.all, "tok") + r.connected() + old := r.conn + r.connected() + r.svc.BoxGone(r.home, "", old) + r.clock.advance(time.Hour) + if n := unreachableSent(t, r); n != 0 { + t.Fatalf("sent %d for a box that was back", n) + } + if away, _ := r.reg.AwayAll(); len(away) != 0 { + t.Fatalf("an absence was recorded: %+v", away) + } +} diff --git a/internal/registry/registry.go b/internal/registry/registry.go index 2c691a1..c9e1d92 100644 --- a/internal/registry/registry.go +++ b/internal/registry/registry.go @@ -9,6 +9,7 @@ import ( "database/sql" "errors" "fmt" + "sync" "time" _ "modernc.org/sqlite" @@ -28,8 +29,16 @@ const ( // Registry is the SQLite file. type Registry struct { db *sql.DB + + // suspended mirrors the suspensions table, so admission, which runs + // under a home's lock, never waits for SQLite. Every write updates it + // after the file. + susMu sync.Mutex + suspended map[suspendKey]uint64 } +type suspendKey struct{ home, device string } + // Home is one registered home. type Home struct { ID string @@ -89,7 +98,40 @@ CREATE TABLE IF NOT EXISTS away ( _ = db.Close() return nil, fmt.Errorf("registry: schema: %w", err) } - return &Registry{db: db}, nil + r := &Registry{db: db, suspended: map[suspendKey]uint64{}} + if err := r.loadSuspensions(); err != nil { + _ = db.Close() + return nil, fmt.Errorf("registry: suspensions: %w", err) + } + return r, nil +} + +func (r *Registry) loadSuspensions() error { + rows, err := r.db.Query(`SELECT home, device_id, epoch FROM suspensions`) + if err != nil { + return err + } + defer rows.Close() + for rows.Next() { + var k suspendKey + var epoch int64 + if err := rows.Scan(&k.home, &k.device, &epoch); err != nil { + return err + } + r.suspended[k] = uint64(epoch) + } + return rows.Err() +} + +// forgetSuspensions drops a home's suspensions from memory, after the file. +func (r *Registry) forgetSuspensions(home string) { + r.susMu.Lock() + defer r.susMu.Unlock() + for k := range r.suspended { + if k.home == home { + delete(r.suspended, k) + } + } } // Close closes the file. @@ -227,7 +269,11 @@ func (r *Registry) Purge(home string, grantID []byte, now time.Time) error { ON CONFLICT (home) DO NOTHING`, home, grantID, now.UnixMilli()); err != nil { return err } - return tx.Commit() + if err := tx.Commit(); err != nil { + return err + } + r.forgetSuspensions(home) + return nil } // Tombstone returns a purged home's grant id and when it was deleted. @@ -278,6 +324,7 @@ func (r *Registry) Expire(now time.Time) ([]string, error) { if _, err := r.db.Exec(`DELETE FROM suspensions WHERE home = ?`, h); err != nil { return nil, err } + r.forgetSuspensions(h) } _, err = r.db.Exec(`DELETE FROM tombstones WHERE deleted_ms < ?`, now.Add(-TombstoneKept).UnixMilli()) return gone, err @@ -288,6 +335,11 @@ func (r *Registry) Expire(now time.Time) ([]string, error) { // which the caller checks, and the suspension ends when the roster drops the // device or its epoch changes. func (r *Registry) Suspend(home, deviceID string, epoch uint64, now time.Time) error { + // Memory first: a suspension narrows, so it bites before the write + // is done, and a write that fails leaves it biting until a restart. + r.susMu.Lock() + r.suspended[suspendKey{home, deviceID}] = epoch + r.susMu.Unlock() _, err := r.db.Exec(`INSERT INTO suspensions (home, device_id, epoch, since_ms) VALUES (?, ?, ?, ?) ON CONFLICT (home, device_id) DO UPDATE SET epoch = excluded.epoch, since_ms = excluded.since_ms`, home, deviceID, int64(epoch), now.UnixMilli()) @@ -296,19 +348,22 @@ func (r *Registry) Suspend(home, deviceID string, epoch uint64, now time.Time) e // Unsuspend lifts a suspension. func (r *Registry) Unsuspend(home, deviceID string) error { - _, err := r.db.Exec(`DELETE FROM suspensions WHERE home = ? AND device_id = ?`, home, deviceID) - return err + if _, err := r.db.Exec(`DELETE FROM suspensions WHERE home = ? AND device_id = ?`, home, deviceID); err != nil { + return err + } + r.susMu.Lock() + delete(r.suspended, suspendKey{home, deviceID}) + r.susMu.Unlock() + return nil } -// Suspended reports whether a device is suspended at this epoch. A read that -// fails counts as suspended: the safe answer narrows. +// Suspended reports whether a device is suspended at this epoch. It reads +// memory, never the file, so admission under a home's lock never waits. func (r *Registry) Suspended(home, deviceID string, epoch uint64) bool { - var held int64 - err := r.db.QueryRow(`SELECT epoch FROM suspensions WHERE home = ? AND device_id = ?`, home, deviceID).Scan(&held) - if errors.Is(err, sql.ErrNoRows) { - return false - } - return err != nil || uint64(held) == epoch + r.susMu.Lock() + defer r.susMu.Unlock() + held, ok := r.suspended[suspendKey{home, deviceID}] + return ok && held == epoch } // PruneSuspensions drops the suspensions a new roster ended: devices it no diff --git a/internal/server/e2e_test.go b/internal/server/e2e_test.go index 2f590fe..9ce63dd 100644 --- a/internal/server/e2e_test.go +++ b/internal/server/e2e_test.go @@ -91,9 +91,9 @@ func newStack(t *testing.T) *stack { tokens.Roster(home, r) } up.OnGone = pushSvc.BoxGone - up.OnConnect = func(home string) { + up.OnConnect = func(home string, conn uint64) { app.BoxConnected(home) - pushSvc.BoxConnected(home) + pushSvc.BoxConnected(home, conn) } ctx, cancel := context.WithCancel(context.Background()) t.Cleanup(cancel) diff --git a/internal/server/server.go b/internal/server/server.go index b21a127..6f78aad 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -144,11 +144,18 @@ func revision() string { func (s *Server) Upkeep(ctx context.Context, every time.Duration) { t := time.NewTicker(every) defer t.Stop() + var tunDropped int64 for { select { case <-ctx.Done(): return case <-t.C: + // Tunnel messages dropped past a home's limits, as a count for + // the whole service; it names no home. + if n := s.Uplink.TunDropped.Load(); n > tunDropped { + s.Log.Warn("tun dropped past homes' limits", "since_last", n-tunDropped, "total", n) + tunDropped = n + } s.Store.Sweep() if err := s.Registry.TokenExpire(time.Now()); err != nil { s.Log.Warn("token upkeep failed", "err", err) diff --git a/internal/server/tunnel_e2e_test.go b/internal/server/tunnel_e2e_test.go index beac4ce..6c74606 100644 --- a/internal/server/tunnel_e2e_test.go +++ b/internal/server/tunnel_e2e_test.go @@ -60,8 +60,15 @@ type boxSide struct { func (s *stack) connectTunBox() *tunBox { s.t.Helper() key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) - pk, _ := cloudwire.PublicKeyBytesOf(&key.PublicKey) static, _ := appwire.GenerateKeyPair() + return s.connectTunBoxAs(key, static) +} + +// connectTunBoxAs is the same box dialling again, as after a network change +// while its old socket is still open. +func (s *stack) connectTunBoxAs(key *ecdsa.PrivateKey, static appwire.KeyPair) *tunBox { + s.t.Helper() + pk, _ := cloudwire.PublicKeyBytesOf(&key.PublicKey) b := &tunBox{t: s.t, key: key, home: cloudwire.HomeID(pk), static: static, allowed: map[string][]byte{}, sessions: map[string]*boxSide{}, purged: make(chan cloudwire.Purged, 4), closed: make(chan struct{})} pool := x509.NewCertPool() @@ -272,6 +279,8 @@ type tunPhone struct { in *appwire.InitiatorState tuns chan cloudwire.AppTun code int + // ended is how the stream ended, set before tuns closes. + ended error } // dialPhone opens the app session. A refusal comes back as code, before @@ -326,6 +335,7 @@ func (s *stack) dialPhone(home string, key appwire.KeyPair, now cloudwire.Now, o _ = conn.SetReadDeadline(time.Time{}) _, msg, err := conn.ReadMessage() if err != nil { + p.ended = err close(p.tuns) return } @@ -698,3 +708,56 @@ func TestOnlyTheWebAppsOriginIsLetIn(t *testing.T) { t.Fatalf("the web app's origin was refused: %d", p.code) } } + +// A box that dials again while its old socket is still open: the phone's +// tunnel on the old uplink is gone at the box, and the phone hears so. +func TestABoxThatRedialsEndsThePhonesOldTunnel(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() + key, _ := appwire.GenerateKeyPair() + box.roster(2, grantOff, nil, key.Public()) + waitRoster(t, s, box.home, key) + phone := s.dialPhone(box.home, key, cloudwire.Now{Hz: 1}, "") + phone.open(box.static.Public(), 11, nil) + + again := s.connectTunBoxAs(box.key, box.static) + end := phone.next() + if end.Sid != 11 || len(end.D) != 0 { + t.Fatalf("after the box redialled the phone got %+v", end) + } + // The new uplink carries a new session. + again.roster(3, grantOff, nil, key.Public()) + time.Sleep(100 * time.Millisecond) + phone.open(again.static.Public(), 12, nil) + if _, res := phone.cmd(12, "0192f2a0-7c1e-7000-8000-00000000b0b1"); res != protocol.CmdApplied { + t.Fatalf("result %q on the new uplink", res) + } +} + +// A phone that sends a d larger than any session message breaks the +// protocol and is closed with 4400. +func TestAnOversizeTunClosesThePhone(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() + key, _ := appwire.GenerateKeyPair() + box.roster(2, grantOff, nil, key.Public()) + waitRoster(t, s, box.home, key) + phone := s.dialPhone(box.home, key, cloudwire.Now{Hz: 1}, "") + phone.sendRaw(cloudwire.AppTun{Sid: 1, D: make([]byte, cloudwire.TunDataMax+1)}) + deadline := time.After(3 * time.Second) + for { + select { + case _, ok := <-phone.tuns: + if ok { + continue + } + var ce *websocket.CloseError + if !asClose(phone.ended, &ce) || ce.Code != cloudwire.CloseProtocol { + t.Fatalf("closed with %v", phone.ended) + } + return + case <-deadline: + t.Fatal("the session stayed open") + } + } +} diff --git a/internal/store/session.go b/internal/store/session.go index fdae437..54b869f 100644 --- a/internal/store/session.go +++ b/internal/store/session.go @@ -58,9 +58,16 @@ func (x *Handle) Open() (Snapshot, *Session, error) { x.h.mu.Lock() defer x.h.mu.Unlock() // The access checked in For may have changed since: check it again - // under the lock that roster changes take. A pairing was checked, and - // spent, there. - if _, _, ok := x.s.access(x.id, x.h, byKey(x.device.Key)); !ok && !x.pairing { + // under the lock that roster changes take. A pairing hash is spent here, + // once the session is sure to open, so a busy home does not burn it. + if x.pairing { + if len(x.h.sessions) >= MaxSessionsPerHome { + return Snapshot{}, nil, ErrBusy + } + if !x.h.spendLocked(x.pair, x.s.now()) { + return Snapshot{}, nil, ErrDenied + } + } else if _, _, ok := x.s.access(x.id, x.h, byKey(x.device.Key)); !ok { return Snapshot{}, nil, ErrDenied } for sess := range x.h.sessions { diff --git a/internal/store/store.go b/internal/store/store.go index 317eae1..49cd462 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -125,6 +125,7 @@ type Handle struct { // runs out at pairedUntil, or the roster lists its key. pairing bool pairedUntil time.Time + pair []byte } // For returns the principal's access to its home, or ErrDenied. A box may @@ -153,7 +154,7 @@ func (s *Store) For(p Principal) (*Handle, error) { return &Handle{s: s, id: p.Home, h: h, kind: Device, device: dev, scopes: scopes}, nil } if until, ok := s.pair(h, p); ok { - return &Handle{s: s, id: p.Home, h: h, kind: Device, pairing: true, pairedUntil: until, + return &Handle{s: s, id: p.Home, h: h, kind: Device, pairing: true, pairedUntil: until, pair: bytes.Clone(p.Pair), device: cloudwire.Device{ID: cloudwire.DeviceID(p.DeviceKey), Key: bytes.Clone(p.DeviceKey), Tun: true}}, nil } return nil, ErrDenied @@ -176,25 +177,42 @@ func newHome() *home { pending: map[string]time.Time{}, spent: map[string]time.Time{}} } -// pair spends a pending pairing hash for a key the roster does not list. The -// device it admits may only open a tunnel; the box decides inside it whether -// the phone pairs. The caller holds h.mu. +// pair checks a pending pairing hash for a key the roster does not list at +// all: a listed key that access refused, suspended or without a tunnel, never +// gets in this way. The device it admits may only open a tunnel; the box +// decides inside it whether the phone pairs. Open spends the hash. The +// caller holds h.mu. func (s *Store) pair(h *home, p Principal) (time.Time, bool) { if len(p.Pair) != cloudwire.PairingHashBytes || len(p.DeviceKey) != 32 || h.roster == nil { return time.Time{}, false } + for _, d := range h.roster.Devices { + if bytes.Equal(d.Key, p.DeviceKey) { + return time.Time{}, false + } + } now := s.now() if now.After(h.rosterAt.Add(time.Duration(h.roster.TTLS) * time.Second)) { return time.Time{}, false } - key := hex.EncodeToString(p.Pair) - until, ok := h.pending[key] + until, ok := h.pending[hex.EncodeToString(p.Pair)] if !ok || !now.Before(until) { return time.Time{}, false } + return until, true +} + +// spendLocked takes a pending pairing hash for good. False means it went +// meanwhile. The caller holds h.mu. +func (h *home) spendLocked(pair []byte, now time.Time) bool { + key := hex.EncodeToString(pair) + until, ok := h.pending[key] + if !ok || !now.Before(until) { + return false + } delete(h.pending, key) h.spent[key] = until - return until, true + return true } // byKey matches the roster entry with a device's Noise static key. @@ -237,9 +255,15 @@ func (s *Store) access(id string, h *home, match func(cloudwire.Device) bool) (c // Connected marks the box's socket up and returns its number. A later // connection replaces this one; Disconnected with an older number is ignored. +// +// A new connection also ends every tunnel the older one carried: the box +// dropped those sessions with its old socket, so their phones hear it now, +// even when the old socket is still open and its own end comes too late to +// count. func (x *Handle) Connected() uint64 { x.h.mu.Lock() defer x.h.mu.Unlock() + x.h.tunGoneLocked() x.h.conn++ x.h.online = true x.h.lastSeen = x.s.now() @@ -320,6 +344,10 @@ func (x *Handle) SetRoster(r *cloudwire.Roster) bool { return false } x.h.roster, x.h.rosterAt = r, x.s.now() + if !slices.Contains(r.Grant.Purposes, protocol.CloudPurposeServe) { + // The copy is off or withdrawn: no reading stays in memory. + x.h.latest, x.h.full = map[string]Held{}, nil + } x.h.setPendingLocked(r, x.h.rosterAt) x.recheckLocked() return true @@ -371,7 +399,7 @@ func (x *Handle) recheckLocked() { case !ok: sess.endLocked(cloudwire.CloseDenied) case dev.ID != sess.x.device.ID || dev.Epoch != sess.x.device.Epoch || dev.Role != sess.x.device.Role || - !slices.Equal(scopes, sess.x.scopes): + dev.Tun != sess.x.device.Tun || !slices.Equal(scopes, sess.x.scopes): sess.endLocked(cloudwire.CloseChanged) } } @@ -515,8 +543,10 @@ func (s *Store) Seed(id string, r *cloudwire.Roster, at time.Time) { if h.roster != nil { return } + // Pairing codes are not restored: which of them a phone already spent + // is not on disk, and the box lists any still on its screen when it + // reconnects. h.roster, h.rosterAt, h.lastSeen = r, at, at - h.setPendingLocked(r, at) } // Forget drops a home the registry has deleted, as after 90 days without a diff --git a/internal/store/tunnel_test.go b/internal/store/tunnel_test.go new file mode 100644 index 0000000..9fd46ed --- /dev/null +++ b/internal/store/tunnel_test.go @@ -0,0 +1,142 @@ +package store + +import ( + "bytes" + "errors" + "testing" + "time" + + "github.com/srcfl/ftw/go/protocol" + "github.com/srcfl/ftw/go/protocol/cloudwire" +) + +var newKey = bytes.Repeat([]byte{0x31}, 32) + +func tunRoster(seq uint64, serve bool, pending ...[]byte) *cloudwire.Roster { + r := &cloudwire.Roster{RosterSeq: seq, TTLS: 86_400, + Devices: []cloudwire.Device{ + {ID: "owner", Key: ownerKey, Role: "owner", Scopes: protocol.CloudScopes, Tun: true}, + {ID: "watch", Key: viewerKey, Role: "watch", Scopes: []string{protocol.ScopeLiveRead}}, + }, + Grant: cloudwire.Grant{Controller: "sourceful"}} + if serve { + r.Grant = cloudwire.Grant{ID: make([]byte, 16), Scopes: []string{protocol.ScopeLiveRead}, + Purposes: []string{protocol.CloudPurposeServe}, Controller: "sourceful"} + } + for _, h := range pending { + r.Pending = append(r.Pending, cloudwire.Pending{Hash: h, TTLS: 600}) + } + return r +} + +func tunHome(t *testing.T, r *cloudwire.Roster) (*Store, *Handle, *clock) { + t.Helper() + c := &clock{t: time.Unix(1_791_820_800, 0)} + s := New(c.now) + box, err := s.For(Principal{Kind: Box, Home: "h1"}) + if err != nil { + t.Fatal(err) + } + box.Connected() + box.SetRoster(r) + return s, box, c +} + +// A box that connects again ends the tunnels its older uplink carried, even +// while the old socket is still open. +func TestANewUplinkEndsTheOldOnesTunnels(t *testing.T) { + s, box, _ := tunHome(t, tunRoster(1, false)) + dev, err := s.For(Principal{Kind: Device, Home: "h1", DeviceKey: ownerKey}) + if err != nil { + t.Fatal(err) + } + _, sess, err := dev.Open() + if err != nil { + t.Fatal(err) + } + sess.SetSid(7) + box.Connected() + select { + case m := <-sess.Tun(): + if m.Sid != 7 || len(m.D) != 0 { + t.Fatalf("the phone heard %+v", m) + } + default: + t.Fatal("the phone's tunnel outlived its uplink") + } +} + +// Readings go when a roster stops serving, however the copy ended. +func TestReadingsGoWhenTheRosterStopsServing(t *testing.T) { + s, box, _ := tunHome(t, tunRoster(1, true)) + _ = box.Put(cloudwire.MsgLive, []byte("live"), 2, true) + box.SetRoster(tunRoster(2, false)) + box.SetRoster(tunRoster(3, true)) + dev, _ := s.For(Principal{Kind: Device, Home: "h1", DeviceKey: ownerKey}) + if snap, _ := dev.Snapshot(); len(snap.Envelopes) != 0 { + t.Fatalf("%d readings outlived a roster that did not serve", len(snap.Envelopes)) + } +} + +// A pairing hash admits only a key the roster does not list, is spent only +// when the session opens, and does not come back with a restart. +func TestAPairingHashIsForANewKeyOnce(t *testing.T) { + hash := bytes.Repeat([]byte{0x77}, cloudwire.PairingHashBytes) + s, _, c := tunHome(t, tunRoster(1, false, hash)) + // The watch is listed, may not tunnel and the copy is off: refused, and + // the hash does not let it in either. + if _, err := s.For(Principal{Kind: Device, Home: "h1", DeviceKey: viewerKey, Pair: hash}); !errors.Is(err, ErrDenied) { + t.Fatalf("a listed key came in by pairing: %v", err) + } + // A busy home spends nothing. + h := s.homes["h1"] + for range MaxSessionsPerHome { + h.sessions[&Session{x: &Handle{device: cloudwire.Device{Key: bytes.Repeat([]byte{9}, 32)}}}] = struct{}{} + } + x, err := s.For(Principal{Kind: Device, Home: "h1", DeviceKey: newKey, Pair: hash}) + if err != nil { + t.Fatal(err) + } + if _, _, err := x.Open(); !errors.Is(err, ErrBusy) { + t.Fatalf("a full home: %v", err) + } + h.sessions = map[*Session]struct{}{} + if _, _, err := x.Open(); err != nil { + t.Fatalf("the hash was spent by a busy home: %v", err) + } + other := bytes.Repeat([]byte{0x32}, 32) + y, err := s.For(Principal{Kind: Device, Home: "h1", DeviceKey: other, Pair: hash}) + if err == nil { + if _, _, err := y.Open(); err == nil { + t.Fatal("a pairing hash opened two sessions") + } + } + + // A restart seeds the roster without its pairing codes. + fresh := New(c.now) + fresh.Seed("h1", tunRoster(1, false, hash), c.now()) + if _, err := fresh.For(Principal{Kind: Device, Home: "h1", DeviceKey: other, Pair: hash}); err == nil { + t.Fatal("a restart brought a pairing code back") + } +} + +// A roster that takes a device's tunnel away ends its session. +func TestLosingTunEndsTheSession(t *testing.T) { + s, box, _ := tunHome(t, tunRoster(1, true)) + dev, _ := s.For(Principal{Kind: Device, Home: "h1", DeviceKey: ownerKey}) + _, sess, err := dev.Open() + if err != nil { + t.Fatal(err) + } + r := tunRoster(2, true) + r.Devices[0].Tun = false + box.SetRoster(r) + select { + case <-sess.Done(): + if sess.Code() != cloudwire.CloseChanged { + t.Fatalf("closed with %d", sess.Code()) + } + default: + t.Fatal("the session kept a tunnel the roster took away") + } +} diff --git a/internal/uplink/hooks_test.go b/internal/uplink/hooks_test.go index 12c07c1..8f5ae77 100644 --- a/internal/uplink/hooks_test.go +++ b/internal/uplink/hooks_test.go @@ -22,7 +22,7 @@ type heard struct { func listen(s *service) *heard { h := &heard{} - s.up.OnConnect = func(string) { h.mu.Lock(); h.conns++; h.mu.Unlock() } + s.up.OnConnect = func(string, uint64) { h.mu.Lock(); h.conns++; h.mu.Unlock() } s.up.OnHeld = func(home, class string, env []byte) { h.mu.Lock() defer h.mu.Unlock() @@ -36,7 +36,7 @@ func listen(s *service) *heard { h.rosters = append(h.rosters, r.RosterSeq) h.mu.Unlock() } - s.up.OnGone = func(home, why string) { + s.up.OnGone = func(home, why string, _ uint64) { h.mu.Lock() h.gone = append(h.gone, why) h.mu.Unlock() diff --git a/internal/uplink/tunnel_test.go b/internal/uplink/tunnel_test.go index d248b78..0a8316e 100644 --- a/internal/uplink/tunnel_test.go +++ b/internal/uplink/tunnel_test.go @@ -203,3 +203,16 @@ func TestPastTheTunLimitsTheServiceDrops(t *testing.T) { t.Fatal("the box is gone") } } + +// A box that sends tun at twice the home's limits is not pacing its tunnel, +// and its uplink closes. +func TestABoxFloodingTunIsClosed(t *testing.T) { + s := newService(t) + b := tunConnect(t, s) + for range 3 * tunMsgBurst { + b.tun(cloudwire.Tun{Dev: "nobody", Sid: 1}) + } + if !b.closed() { + t.Fatal("a box flooding tun stayed connected") + } +} diff --git a/internal/uplink/uplink.go b/internal/uplink/uplink.go index a93c3c1..c82f913 100644 --- a/internal/uplink/uplink.go +++ b/internal/uplink/uplink.go @@ -49,13 +49,14 @@ const ( // is counted apart. rateBurst = 60 ratePerSec = 10 - // tun per home and direction, as the relay allows one stream. Past it - // the service drops tun and counts the drop; Noise transport survives a - // lost message. - tunMsgBurst = 240 - tunMsgPerSec = 60 - tunByteBurst = 4 << 20 - tunBytePerSec = 512 << 10 + // tun per home and direction, the limits the relay puts on a box's + // uplink, shared with the box, which paces its bulk under them. Past + // them the service drops tun and counts the drop; Noise transport + // survives a lost message. + tunMsgBurst = cloudwire.TunMsgBurst + tunMsgPerSec = cloudwire.TunMsgsPerSecond + tunByteBurst = cloudwire.TunByteBurst + tunBytePerSec = cloudwire.TunBytesPerSecond // maxPending bounds the pairing codes one roster may list, and // maxPendingTTL how long one may last. maxPending = 4 @@ -81,8 +82,10 @@ type Uplink struct { // OnConnect runs after a box's hello_ok, so the app sessions can ask a // box that just came back for one-second updates and push can stop - // waiting to say it is out of reach. Set before serving. - OnConnect func(home string) + // waiting to say it is out of reach. conn, here and in OnGone, is the + // home's connection number, which rises with each connection, so a + // hook heard late can be told from a newer one. Set before serving. + OnConnect func(home string, conn uint64) // OnHeld gets each envelope the service holds, with its class, as the // box signed it; OnRoster each roster the service keeps; OnGone the end // of the home's current uplink, with the reason the box gave in its @@ -90,7 +93,7 @@ type Uplink struct { // the box's goroutine and must not block. Set before serving. OnHeld func(home, class string, envelope []byte) OnRoster func(home string, r *cloudwire.Roster) - OnGone func(home, why string) + OnGone func(home, why string, conn uint64) } // New builds the handler. key is the service key that signs hello_ok and @@ -131,7 +134,9 @@ func (u *Uplink) ServeHTTP(w http.ResponseWriter, r *http.Request) { now := u.now() c := &boxConn{u: u, conn: conn, ekm: ekm, tokens: rateBurst, refilled: now, upMsgs: newBucket(tunMsgBurst, tunMsgPerSec, now), upBytes: newBucket(tunByteBurst, tunBytePerSec, now), - downMsgs: newBucket(tunMsgBurst, tunMsgPerSec, now), downBytes: newBucket(tunByteBurst, tunBytePerSec, now)} + upHardMsgs: newBucket(2*tunMsgBurst, 2*tunMsgPerSec, now), + upHardBytes: newBucket(2*tunByteBurst, 2*tunBytePerSec, now), + downMsgs: newBucket(tunMsgBurst, tunMsgPerSec, now), downBytes: newBucket(tunByteBurst, tunBytePerSec, now)} c.write = c.writeFrame if err := c.run(); err != nil && !websocket.IsCloseError(err, websocket.CloseNormalClosure, websocket.CloseGoingAway) { u.log.Debug("box uplink closed", "err", err) @@ -224,8 +229,11 @@ type boxConn struct { refilled time.Time // The tunnel's limits: up from the box, read on its goroutine; down - // from the home's phones, from theirs. + // from the home's phones, from theirs. upHard is twice the up limits: a + // box past it is not pacing its tunnel, and its uplink closes. upMsgs, upBytes bucket + upHardMsgs bucket + upHardBytes bucket downMu sync.Mutex downMsgs bucket downBytes bucket @@ -334,11 +342,11 @@ func (c *boxConn) run() error { } c.u.mu.Unlock() if current && c.u.OnGone != nil { - c.u.OnGone(c.home, c.bye) + c.u.OnGone(c.home, c.bye, gen) } }() if c.u.OnConnect != nil { - go c.u.OnConnect(c.home) + go c.u.OnConnect(c.home, gen) } } @@ -404,7 +412,11 @@ func (c *boxConn) fromBox(env cloudwire.Envelope) error { if len(env.Sig) != 0 || cloudwire.Unmarshal(env.B, &t) != nil || len(t.D) > cloudwire.TunDataMax || t.Dev == "" { return fmt.Errorf("%w: a tun the service cannot route", errProtocol) } - if !tunAllowed(&c.upMsgs, &c.upBytes, len(t.D), c.u.now()) { + now := c.u.now() + if !tunAllowed(&c.upHardMsgs, &c.upHardBytes, len(t.D), now) { + return fmt.Errorf("%w: tun at twice the home's limits", errProtocol) + } + if !tunAllowed(&c.upMsgs, &c.upBytes, len(t.D), now) { c.u.TunDropped.Add(1) return nil } diff --git a/internal/uplink/uplink_test.go b/internal/uplink/uplink_test.go index eaa9a2c..1321708 100644 --- a/internal/uplink/uplink_test.go +++ b/internal/uplink/uplink_test.go @@ -403,7 +403,8 @@ func FuzzBoxTun(f *testing.F) { now := time.Now() c := &boxConn{u: u, home: home, pub: &key.PublicKey, handle: handle, tun: true, write: func([]byte) error { return nil }, - upMsgs: newBucket(tunMsgBurst, tunMsgPerSec, now), upBytes: newBucket(tunByteBurst, tunBytePerSec, now)} + upMsgs: newBucket(tunMsgBurst, tunMsgPerSec, now), upBytes: newBucket(tunByteBurst, tunBytePerSec, now), + upHardMsgs: newBucket(2*tunMsgBurst, 2*tunMsgPerSec, now), upHardBytes: newBucket(2*tunByteBurst, 2*tunBytePerSec, now)} _ = c.fromBox(cloudwire.Envelope{T: cloudwire.MsgTun, B: body}) }) } From e897a34c2b6b0184b246c7804470891dcf548f1e Mon Sep 17 00:00:00 2001 From: Fredrik Ahlgren Date: Thu, 15 Oct 2026 17:46:38 +0200 Subject: [PATCH 3/4] test: the readable copy turns on and off on a live uplink Review of srcfl/ftw#1635, point 4: hello's caps and grantId describe the connection's start; the service takes what it may hold from the newest roster's grant. The README says so. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Fredrik Ahlgren --- README.md | 2 ++ internal/server/tunnel_e2e_test.go | 54 ++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/README.md b/README.md index 3c66572..6dc9eb4 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,8 @@ to it yet. - Each home's newest message of each class lives in memory, as the exact bytes the box sent. Readings are dropped an hour after the box goes quiet and are never written to disk. +- What the service holds follows the newest roster's grant, never `hello`: + the readable copy can turn on or off while the uplink stays up. - The registry (SQLite) holds the home id, the box key, the signed roster, owners' suspensions, APNs registrations, the hashes of extension tokens and when an away box's alert is due. A box whose owner turns the readable diff --git a/internal/server/tunnel_e2e_test.go b/internal/server/tunnel_e2e_test.go index 6c74606..19b760c 100644 --- a/internal/server/tunnel_e2e_test.go +++ b/internal/server/tunnel_e2e_test.go @@ -761,3 +761,57 @@ func TestAnOversizeTunClosesThePhone(t *testing.T) { } } } + +// The readable copy turns on and off while the box's uplink stays up: the +// service takes what it may hold from the newest roster's grant, not from +// hello, which named no grant. off → on → off on one connection. +func TestTheCopyTurnsOnAndOffMidConnection(t *testing.T) { + s := newStack(t) + box := s.connectTunBox() // hello names no grant + key, _ := appwire.GenerateKeyPair() + box.roster(2, grantOff, nil, key.Public()) + waitRoster(t, s, box.home, key) + live := func(w int64) { + box.send(cloudwire.MsgLive, cloudwire.Live{Header: box.header(), Fields: map[string]int64{"2": w}, Ages: map[string]int64{}}) + } + held := func() int { + p := s.dialPhone(box.home, key, cloudwire.Now{}, "") + if p.conn != nil { + _ = p.conn.Close() + } + return len(p.snap.Envelopes) + } + + live(1) + time.Sleep(100 * time.Millisecond) + if n := held(); n != 0 { + t.Fatalf("off: %d envelopes held", n) + } + + grantID := bytes.Repeat([]byte{8}, cloudwire.GrantIDBytes) + box.roster(3, cloudwire.Grant{ID: grantID, Scopes: []string{protocol.ScopeLiveRead}, + Purposes: []string{protocol.CloudPurposeServe}, Controller: cloudwire.ControllerSourceful}, nil, key.Public()) + live(2) + eventually(t, "on: the copy is held on the same connection", func() bool { return held() == 1 }) + + box.roster(4, cloudwire.Grant{ID: grantID, Scopes: []string{}, Purposes: []string{}, + Controller: cloudwire.ControllerSourceful}, nil, key.Public()) + select { + case p := <-box.purged: + if p.What != cloudwire.PurgedCopy { + t.Fatalf("receipt %+v", p) + } + case <-time.After(3 * time.Second): + t.Fatal("no receipt") + } + live(3) + time.Sleep(100 * time.Millisecond) + if n := held(); n != 0 { + t.Fatalf("off again: %d envelopes held", n) + } + select { + case <-box.closed: + t.Fatal("the uplink closed while the copy turned on and off") + default: + } +} From fce0e8f81b3893da405540ea2308f904dbe704d7 Mon Sep 17 00:00:00 2001 From: Fredrik Ahlgren Date: Thu, 15 Oct 2026 18:44:03 +0200 Subject: [PATCH 4/4] chore: pin go/protocol to srcfl/ftw master after #1635 (f5cdb920) Co-Authored-By: Claude Opus 5.5 --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 17a2003..fbaf99f 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.26.0 require ( github.com/fxamacker/cbor/v2 v2.9.4 github.com/gorilla/websocket v1.5.3 - github.com/srcfl/ftw/go/protocol v0.0.0-20261015152627-0970e7b36690 + github.com/srcfl/ftw/go/protocol v0.0.0-20261015164244-f5cdb92058c1 golang.org/x/crypto v0.57.0 golang.org/x/sys v0.48.0 modernc.org/sqlite v1.59.0 diff --git a/go.sum b/go.sum index 4439303..83b0615 100644 --- a/go.sum +++ b/go.sum @@ -16,8 +16,8 @@ github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOF github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= -github.com/srcfl/ftw/go/protocol v0.0.0-20261015152627-0970e7b36690 h1:wnkUzAoSXPBuosc2awje2vZdUQdTwiXVYWwfOZZ5P7w= -github.com/srcfl/ftw/go/protocol v0.0.0-20261015152627-0970e7b36690/go.mod h1:VEjr/o0yZijJ1nApuJK+nak6FxJ/fubAUBNOunqNrUU= +github.com/srcfl/ftw/go/protocol v0.0.0-20261015164244-f5cdb92058c1 h1:JLItMdT1tX3ejgmW80VXcwZ77wBPY7WT2ewMGfPLDIc= +github.com/srcfl/ftw/go/protocol v0.0.0-20261015164244-f5cdb92058c1/go.mod h1:VEjr/o0yZijJ1nApuJK+nak6FxJ/fubAUBNOunqNrUU= github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=