You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Status after the member ladder runs on the bananas testnet (4 routers, high watermark 4000 inbound each; members are agentmesh-node processes on 10 GCE VMs, driven by tests/scale/member-ladder.sh).
Tier
Result
What stopped it
1000
Reached twice. Run 3 (2026-10-10, main at 6bcf830): 500/500 then 1000/1000 resident, zero exits.
—
5000
Not reached. Run 2: 1432 resident. Run 3: 3248 resident.
Run 2: control plane enrollment limiter (global 10/s) answered 429 and the node treated it as fatal; router CPU limits (500m) throttled handshakes. Run 3: the 429 retry gave up after ~20 s where a fleet of 1500 joining at 10/s needs 150 s; and every member attached to every router, so the four routers sat at 3200–3500 of 4000 inbound and the next step would have trimmed the whole mesh.
10k
Not attempted.
Same bound: with every member on every router, adding routers does not add members.
What the 3248-member run did show: all members survived a control plane and router rollout mid-hold; routers at 455–585 MiB RSS, ~20k goroutines, 335–440 mCPU each; control plane 135–180 MiB; a node at 59 MiB, 336 goroutines, 15 fds.
Fixed so far: #615 (router graceful shutdown, relay budget, router metrics, --reachability), #638 (enrollment retries 429 with Retry-After, node retries router admission instead of exiting, router resource limits sized from the watermarks, 502 for an unreachable provider, CPU limits removed from the testnet manifests).
In preparation, one PR: the enrollment retry is bounded by time (3 m) rather than attempts; a trim sheds 15% of the connections instead of 75%; /info lists every router with its labels and load; a node attaches to 2 routers (--routers), chosen by --router-selector/--router-prefer and load, authenticates on demand to a router it transits for a circuit, and publishes a circuit address for the routers it reserved on; a router tells its members to go elsewhere when it drains or is overloaded; and a scalability guide that states the sizing rule (members per router ≈ F × K / N).
Next round, once that lands on bananas: rerun the ladder to 5000 with 4 routers (expected ≈2500 inbound per router), then scale to 6 routers for the 10k tier. What to measure that run 3 could not: per-router inbound tracks F × K / N rather than F; a router restart during the hold moves its members to the others and they top back up; enrollment at 100 joiners/s completes within the retry budget; and the go-away path under drain. This issue closes with those numbers, not with the PR.
Thanks for the detailed ladder write-up. I'd like to help with one narrow piece that doesn't overlap the router work.
On main (cf1d27f) the time-bounded enrollment retry has landed (a888f2a, DefaultEnrollBackoff in internal/controlplane/client/retry.go), and so have the router go-away/trim changes (0b00c60, 2529ff8, covered in internal/router/goaway_test.go). As far as I can tell, nothing tests the sizing claim in the DefaultEnrollBackoff comment: about 1500 members starting at once get through a limiter of EnrollRateLimit = 10/s, burst 20 (internal/controlplane/server.go:97-98), within the 3 min budget, and the retry jitter spreads them out after the Retry-After from tooManyRequests rather than having them come back in lockstep. TestDoWithChallengeRetry covers a single caller. The server test only checks that a 429 carries Retry-After.
What I have in mind is a unit test in internal/controlplane/client: a real rate.Limiter behind an httptest server answering with tooManyRequests, N concurrent callers on DoWithChallengeRetry with an injected clock, and assertions that everyone is admitted within Budget for N ≤ R × Budget and that the first refusal past that is the one returned to the caller. It would be standard testing, table-driven, no new deps, and it would run in well under a second. It would also give the "100 joiners/s within the retry budget" check from your next round a lowest-level counterpart.
Would that be useful, or would you rather I wait until your remaining PR (multi-router attach, /info) lands and pick up something from the next ladder run instead?
Status after the 2026-10-12 ladder run on bananas (main at cf1d27f; 6 routers; members are agentmesh-node processes on 10 GCE VMs, 1000 per VM, driven by tests/scale/member-ladder.sh; every joiner calls a provider through the mesh once it is ready).
step
resident before
joining
ready
first call ok
calls/member
ready p50/p95 ms
1
0
1000
1000
1000
1.38
411/666
2
1000
1000
1000
1000
1.35
410/665
3
2000
1000
1000
1000
1.01
411/1044
4
3000
1000
1000
1000
1.03
664/1046
5
4000
1000
1000
1000
1.00
665/3551
6
5000
1000
1000
1000
1.00
665/1547
7
6000
1000
1000
1000
1.00
666/3049
8
7000
1000
1000
1000
1.01
1547/15067
9
8000
1000
1000
1000
1.02
2549/14069
10
9000
1000
1000
1000
1.01
3549/5552
10,000 members resident, held for 41 minutes: every VM reports 1000 ready for the whole hold, 0 flaps, 0 exits. The ~800 failed call attempts across the whole run (384 in step 1, 349 in step 2, 9 and 28 later) are the provider's connection trim, retried within seconds. Control plane p99 stayed at or under 2.5 s through the 10k step. The routers refused nothing: no connection, stream or memory refusal counter moved at any step.
Routers at 10k: 8,800–9,250 connections each for 3,200–5,100 authenticated members, RSS 1.6–1.74 GiB, 45–65k goroutines. The three tiers in the issue title are reached.
Fixed to get here, each found from the counters the previous one added: #667, #669 (per-protocol stream budgets and refusal counters; the DHT routing table collapsed under the libp2p default of 768 streams per protocol), #685 (per-source connection limits charged to the real source instead of the relay; relay stop-stream budget; QUIC's 256 streams per connection, so the node dials TCP first; --pprof-addr; the scalability guide now describes connections, streams and roles), #687 (resource manager memory: yamux reserves 256 KiB per stream and libp2p's RAM-scaled system budget filled at ~1500 members; stream budgets now carry their memory).
Two things remain, neither blocks the tiers:
Routers keep memory after members leave. Five hours after the drain a router sits at ~720 MiB RSS with 272 MB live heap at 16 connections. A heap profile through the new --pprof-addr attributes ~205 MB of it to two upstream bugs, both with fixes pending: fix: release the writer context when a peer's writer exits libp2p/go-libp2p-pubsub#735 (gossipsub never cancels a dead peer's writer context; one context per peer ever seen, ~150 MB here) and fix(pstoremem): expire addrs downgraded from a connected TTL libp2p/go-libp2p#3559 (the in-memory peerstore never expires an address downgraded from a connected TTL, which is what every disconnect does; 135k addresses and 11k signed peer records, ~55 MB here). We pick them up when they are released.
Connections per router are 2.5× the members it serves. Every member's DHT client connects to every router for lookups, the router closes those connections at the auth deadline, and the next lookup reopens them. It is what drives the two leaks above and the goroutine count. The design for this is separate work.
Still open from this round: #670 (the JS SDK did not advertise its circuit addresses to the routers it reserved on; identify push).
@asivanadi0 Yes, please go ahead, that would be useful: the ladder joins members at 5/s per VM so it never exercised the 429 path, and the claim in the DefaultEnrollBackoff comment has not been observed yet. Keep it fast , standard testing, table-driven, under a second, no new dependencies
Small wrinkle: the wait in DoWithChallengeRetry is a real time.After, and tooManyRequests names whole seconds, so a sub-second test needs an unexported sleep hook in retry.go. OK to include that, or would you rather keep it test-only?
Status after the ladder runs of 2026-10-14/15 on bananas (12 routers, high watermark 6000; members are agentmesh-node processes on 10 GCE VMs, 1000 per VM, driven by tests/scale/member-ladder.sh; every joiner calls a provider through the mesh once it is ready). Four 10k runs, each on main at the commit named, so the fixes between them are measured on the same shape.
L15 and L16, the runs after the directory hardening (#719), answered the question left from the previous update about noisy connections: routers hold connected ≈ authenticated, no unauthenticated streams, no shedding, and L16 held 10,000 for 90 minutes with 0 flaps and 0 exits. What L16 showed next was in the control plane: p99 at 2.5 s, 401s and 500s to members, from a nodes table of 118k rows that nothing deleted, with the metrics scrape reading every row every 30 s.
Storage. A table-by-table audit of the testnet database found every table that grows with enrollment growing without bound: nodes 19,751 (99.2% lapsed), enrollment_requests 99,990, bootstrap_tokens 325 for 9 live, routers 290 leases for 13 live, and one in-memory service report per member that ever reported. #701 added the node GC for lapsed sessions and the counted node metrics; #728 collects bootstrap members that stop refreshing (--node-retention, 30 d) and members of an ephemeral token after an hour, sweeps requests, tokens and leases on the same schedule, prunes the catalog cache, and exports agentmesh_control_plane_storage_rows{table} and the deletion counters so a reaper that stops is visible; #729 moves the cluster's service account issuer to --workload-issuer (the probes had been enrolling as humans with 90-day sessions) and sets --node-retention=72h on bananas. The ladder mints an ephemeral token, and L17 was the validation:
bananas storage
before rollout
pre-ladder
post-ladder
+70 min
nodes
19,870
19,672
29,717
19,748
of which ephemeral (the fleet)
—
0
10,000
0
enrollment_requests
99,990
39,621
49,621
39,620
bootstrap_tokens
325
48
46
46
routers
290
35
35
35
catalog entries in memory
—
28
6,510
34
The first sweep after the rollout removed 60,371 requests, 279 tokens, 257 leases and 283 node rows in five seconds; the sweep an hour after the ladder ended removed the 10,000 ephemeral rows, their 10,001 requests and 6,450 catalog entries (nodes_deleted_total{reason="ephemeral"} 10000). The 19,373 rows still in nodes are probe rows I marked as lapsed sessions, due to go on 2026-10-18 after the 72 h retention; that is the last reaper to observe on real data.
Bugs the runs surfaced, fixed in #730 (L18 is the run after it):
Key rotation never happened on a control plane that restarts more often than --key-rotation-interval: the ticker ran from process start, and bananas redeploys on every merge, so it had not rotated since 2026-10-09. The deadline now lives in the store and is checked at least once a minute; next_key_rotation_timestamp_seconds shows it. On the rollout the overdue window was claimed 51 s after the pods started, the next cold-path probe passed under the new key, and L18 ran on the rotated mesh.
After a rotation, a node showed its refreshed credential to a router that had not synced the new key yet, got refused, dropped the router and answered 503 to every call until the connection monitor's next minute. The re-admission is retried over ~3.5 s first. This was also the flake in TestKeyRotationIntegration.
The 392 exits in L17 were the bench giving each member a fixed loopback port (20000+i) for /readyz, which the kernel's ephemeral range on the VMs covered. The sidecar's /readyz had been answering 200 whether or not the node held a router; it now gives the metrics listener's answer, and the bench reads it over the member's Unix socket, so a member binds no port.
The metrics snapshot listed every enrollment request (tens of thousands) every 30 s; it is a COUNT … GROUP BY now, as for nodes.
A node enrolled with no last-seen instant was stored at the zero time, which the new sweep read as a member silent for two thousand years; sixteen unit tests enrolled nodes that way and one of them lost the race in CI. An unset instant is recorded as now.
What remains, none of it blocking the tiers:
Watch the grace end of the rotation (2026-10-17 21:05Z, key 52 retires) and the +72 h sweep (2026-10-18 ~14:30Z) on bananas.
One stale developer node (a one-use token from 10-13, row gone) retries /refresh every 30 s and fails router handshakes; correct behaviour, loud. It is the 401s in the L17/L18 columns.
The two libp2p upstream leaks and the "connections per router are 2.5× members" design item from the previous update are unchanged.
Activity
aojea commentedon Oct 10, 2026
Status after the member ladder runs on the
bananastestnet (4 routers, high watermark 4000 inbound each; members areagentmesh-nodeprocesses on 10 GCE VMs, driven bytests/scale/member-ladder.sh).What the 3248-member run did show: all members survived a control plane and router rollout mid-hold; routers at 455–585 MiB RSS, ~20k goroutines, 335–440 mCPU each; control plane 135–180 MiB; a node at 59 MiB, 336 goroutines, 15 fds.
Fixed so far: #615 (router graceful shutdown, relay budget, router metrics,
--reachability), #638 (enrollment retries 429 withRetry-After, node retries router admission instead of exiting, router resource limits sized from the watermarks, 502 for an unreachable provider, CPU limits removed from the testnet manifests).In preparation, one PR: the enrollment retry is bounded by time (3 m) rather than attempts; a trim sheds 15% of the connections instead of 75%;
/infolists every router with its labels and load; a node attaches to 2 routers (--routers), chosen by--router-selector/--router-preferand load, authenticates on demand to a router it transits for a circuit, and publishes a circuit address for the routers it reserved on; a router tells its members to go elsewhere when it drains or is overloaded; and a scalability guide that states the sizing rule (members per router ≈ F × K / N).Next round, once that lands on
bananas: rerun the ladder to 5000 with 4 routers (expected ≈2500 inbound per router), then scale to 6 routers for the 10k tier. What to measure that run 3 could not: per-router inbound tracks F × K / N rather than F; a router restart during the hold moves its members to the others and they top back up; enrollment at 100 joiners/s completes within the retry budget; and the go-away path under drain. This issue closes with those numbers, not with the PR.asivanadi0 commentedon Oct 13, 2026
Thanks for the detailed ladder write-up. I'd like to help with one narrow piece that doesn't overlap the router work.
On main (cf1d27f) the time-bounded enrollment retry has landed (a888f2a,
DefaultEnrollBackoffininternal/controlplane/client/retry.go), and so have the router go-away/trim changes (0b00c60, 2529ff8, covered ininternal/router/goaway_test.go). As far as I can tell, nothing tests the sizing claim in theDefaultEnrollBackoffcomment: about 1500 members starting at once get through a limiter ofEnrollRateLimit= 10/s, burst 20 (internal/controlplane/server.go:97-98), within the 3 min budget, and the retry jitter spreads them out after theRetry-AfterfromtooManyRequestsrather than having them come back in lockstep.TestDoWithChallengeRetrycovers a single caller. The server test only checks that a 429 carriesRetry-After.What I have in mind is a unit test in
internal/controlplane/client: a realrate.Limiterbehind anhttptestserver answering withtooManyRequests, N concurrent callers onDoWithChallengeRetrywith an injected clock, and assertions that everyone is admitted withinBudgetfor N ≤ R × Budget and that the first refusal past that is the one returned to the caller. It would be standardtesting, table-driven, no new deps, and it would run in well under a second. It would also give the "100 joiners/s within the retry budget" check from your next round a lowest-level counterpart.Would that be useful, or would you rather I wait until your remaining PR (multi-router attach,
/info) lands and pick up something from the next ladder run instead?aojea commentedon Oct 13, 2026
Status after the 2026-10-12 ladder run on
bananas(main at cf1d27f; 6 routers; members areagentmesh-nodeprocesses on 10 GCE VMs, 1000 per VM, driven bytests/scale/member-ladder.sh; every joiner calls a provider through the mesh once it is ready).10,000 members resident, held for 41 minutes: every VM reports 1000 ready for the whole hold, 0 flaps, 0 exits. The ~800 failed call attempts across the whole run (384 in step 1, 349 in step 2, 9 and 28 later) are the provider's connection trim, retried within seconds. Control plane p99 stayed at or under 2.5 s through the 10k step. The routers refused nothing: no connection, stream or memory refusal counter moved at any step.
Routers at 10k: 8,800–9,250 connections each for 3,200–5,100 authenticated members, RSS 1.6–1.74 GiB, 45–65k goroutines. The three tiers in the issue title are reached.
Fixed to get here, each found from the counters the previous one added: #667, #669 (per-protocol stream budgets and refusal counters; the DHT routing table collapsed under the libp2p default of 768 streams per protocol), #685 (per-source connection limits charged to the real source instead of the relay; relay stop-stream budget; QUIC's 256 streams per connection, so the node dials TCP first;
--pprof-addr; the scalability guide now describes connections, streams and roles), #687 (resource manager memory: yamux reserves 256 KiB per stream and libp2p's RAM-scaled system budget filled at ~1500 members; stream budgets now carry their memory).Two things remain, neither blocks the tiers:
--pprof-addrattributes ~205 MB of it to two upstream bugs, both with fixes pending: fix: release the writer context when a peer's writer exits libp2p/go-libp2p-pubsub#735 (gossipsub never cancels a dead peer's writer context; one context per peer ever seen, ~150 MB here) and fix(pstoremem): expire addrs downgraded from a connected TTL libp2p/go-libp2p#3559 (the in-memory peerstore never expires an address downgraded from a connected TTL, which is what every disconnect does; 135k addresses and 11k signed peer records, ~55 MB here). We pick them up when they are released.Still open from this round: #670 (the JS SDK did not advertise its circuit addresses to the routers it reserved on; identify push).
aojea commentedon Oct 13, 2026
@asivanadi0 Yes, please go ahead, that would be useful: the ladder joins members at 5/s per VM so it never exercised the 429 path, and the claim in the DefaultEnrollBackoff comment has not been observed yet. Keep it fast , standard testing, table-driven, under a second, no new dependencies
asivanadi0 commentedon Oct 13, 2026
Small wrinkle: the wait in
DoWithChallengeRetryis a realtime.After, andtooManyRequestsnames whole seconds, so a sub-second test needs an unexported sleep hook in retry.go. OK to include that, or would you rather keep it test-only?node: stop refreshing the DHT table; the catalog prunes it
node, router: the mesh's replication factor is kademlia's k on a member
node: stop refreshing the DHT table; the catalog prunes it
node, router: the mesh's replication factor is kademlia's k on a member
control plane client: test a fleet through the enrollment limiter
aojea commentedon Oct 15, 2026
Status after the ladder runs of 2026-10-14/15 on
bananas(12 routers, high watermark 6000; members areagentmesh-nodeprocesses on 10 GCE VMs, 1000 per VM, driven bytests/scale/member-ladder.sh; every joiner calls a provider through the mesh once it is ready). Four 10k runs, each onmainat the commit named, so the fixes between them are measured on the same shape.nodestable at the endL15 and L16, the runs after the directory hardening (#719), answered the question left from the previous update about noisy connections: routers hold connected ≈ authenticated, no unauthenticated streams, no shedding, and L16 held 10,000 for 90 minutes with 0 flaps and 0 exits. What L16 showed next was in the control plane: p99 at 2.5 s, 401s and 500s to members, from a
nodestable of 118k rows that nothing deleted, with the metrics scrape reading every row every 30 s.Storage. A table-by-table audit of the testnet database found every table that grows with enrollment growing without bound:
nodes19,751 (99.2% lapsed),enrollment_requests99,990,bootstrap_tokens325 for 9 live,routers290 leases for 13 live, and one in-memory service report per member that ever reported. #701 added the node GC for lapsed sessions and the counted node metrics; #728 collects bootstrap members that stop refreshing (--node-retention, 30 d) and members of an ephemeral token after an hour, sweeps requests, tokens and leases on the same schedule, prunes the catalog cache, and exportsagentmesh_control_plane_storage_rows{table}and the deletion counters so a reaper that stops is visible; #729 moves the cluster's service account issuer to--workload-issuer(the probes had been enrolling as humans with 90-day sessions) and sets--node-retention=72honbananas. The ladder mints an ephemeral token, and L17 was the validation:nodesenrollment_requestsbootstrap_tokensroutersThe first sweep after the rollout removed 60,371 requests, 279 tokens, 257 leases and 283 node rows in five seconds; the sweep an hour after the ladder ended removed the 10,000 ephemeral rows, their 10,001 requests and 6,450 catalog entries (
nodes_deleted_total{reason="ephemeral"} 10000). The 19,373 rows still innodesare probe rows I marked as lapsed sessions, due to go on 2026-10-18 after the 72 h retention; that is the last reaper to observe on real data.Bugs the runs surfaced, fixed in #730 (L18 is the run after it):
--key-rotation-interval: the ticker ran from process start, andbananasredeploys on every merge, so it had not rotated since 2026-10-09. The deadline now lives in the store and is checked at least once a minute;next_key_rotation_timestamp_secondsshows it. On the rollout the overdue window was claimed 51 s after the pods started, the next cold-path probe passed under the new key, and L18 ran on the rotated mesh.TestKeyRotationIntegration./readyz, which the kernel's ephemeral range on the VMs covered. The sidecar's/readyzhad been answering 200 whether or not the node held a router; it now gives the metrics listener's answer, and the bench reads it over the member's Unix socket, so a member binds no port.COUNT … GROUP BYnow, as for nodes.What remains, none of it blocking the tiers:
bananas./refreshevery 30 s and fails router handshakes; correct behaviour, loud. It is the 401s in the L17/L18 columns.