Repository navigation
Member directory - #693
Member directory#693
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces configurable DHT replication and reprovide intervals for nodes, allowing better control over how service records are published and looked up across routers. It also establishes authenticated directory sessions with unattached routers to reuse connections, and prunes the DHT routing table when routers leave the catalog. The review feedback suggests optimizing lock contention and reducing memory allocations in internal/node/directory.go by consolidating lock acquisitions and avoiding temporary map allocations in authDirectoryRouter and DirectoryRouters.
| func (n *AgentMeshNode) authDirectoryRouter(ctx context.Context, router peer.ID) { | ||
| if n.isAttached(router) || n.isAuthenticatedAndConnected(router) || !n.isCatalogRouter(router) { | ||
| return | ||
| } |
There was a problem hiding this comment.
6. Behavior must be auditable from logs and metrics / Efficiency
The authDirectoryRouter function currently performs multiple sequential lock acquisitions on n.mu by calling n.isAttached(router), n.isAuthenticatedAndConnected(router), and n.isCatalogRouter(router) back-to-back. Each of these helper functions acquires and releases the lock individually, leading to unnecessary lock contention and overhead.
We can optimize this by acquiring the lock once, extracting the required state, and performing the checks.
func (n *AgentMeshNode) authDirectoryRouter(ctx context.Context, router peer.ID) {
n.mu.Lock()
attached := n.attached[router]
authenticated := n.authenticatedRouters[router]
isCatalog := false
for _, c := range n.routerCatalog {
if c.ID == router {
isCatalog = true
break
}
}
n.mu.Unlock()
if attached || (authenticated && n.Host.Network().Connectedness(router) == network.Connected) || !isCatalog {
return
}References
- The style guide emphasizes clean, robust, and maintainable Go code. Minimizing lock contention and avoiding unnecessary allocations in hot paths like connection callbacks aligns with these principles. (link)
| func (n *AgentMeshNode) DirectoryRouters() int { | ||
| attached := map[peer.ID]bool{} | ||
| n.mu.Lock() | ||
| for r := range n.attached { | ||
| attached[r] = true | ||
| } | ||
| n.mu.Unlock() | ||
| count := 0 | ||
| for _, r := range n.authenticatedRouterIDs() { | ||
| if !attached[r] { | ||
| count++ | ||
| } | ||
| } | ||
| return count | ||
| } |
There was a problem hiding this comment.
6. Behavior must be auditable from logs and metrics / Efficiency
The DirectoryRouters function currently allocates a temporary map attached and calls n.authenticatedRouterIDs(), which locks n.mu again and allocates a temporary slice of peer IDs. This results in multiple lock acquisitions and unnecessary memory allocations.
We can optimize this by locking n.mu once and performing the checks directly on the live maps.
func (n *AgentMeshNode) DirectoryRouters() int {
n.mu.Lock()
defer n.mu.Unlock()
count := 0
for r := range n.authenticatedRouters {
if !n.attached[r] && n.Host.Network().Connectedness(r) == network.Connected {
count++
}
}
return count
}References
- The style guide emphasizes clean, robust, and maintainable Go code. Minimizing lock contention and avoiding unnecessary allocations in hot paths like connection callbacks aligns with these principles. (link)
The node ran its DHT in ModeAuto, so a member AutoNAT found publicly reachable became a DHT server: listed in every other member's routing table, taking their lookups on a 400-connection budget, answering whoever connected, and trusted for unsigned provider records. The routers are the directory (the roles table says so, the JS SDK is clientMode, the Python SDK has no server), and nothing in the mesh needs a member to serve it. Fix the mode to client. --reachability keeps deciding how a member is reached, direct or through a relay, and nothing else. The DHT options move to dhtOptions so the test can build a member DHT on a host with forced public reachability and check it stays a client while a ModeAuto DHT on the same setup turns server.
A member's kad-dht client kept a connection to every router it was not attached to about four fifths of the time, and at 10,000 members each router accepted 3,900 to 5,100 inbound connections a minute, 99.9% of them closed at the auth deadline without a handshake (google#277). The driver is kad-dht's table maintenance, not lookups: a table of ten peers or fewer counts as starved, so fixLowPeers triggers a refresh every two minutes, and every refresh pings each entry and runs a lookup for the member's own ID against all of them. With six routers as the only servers that is a dial to every other router, closed a minute later, opened again a minute after that. The refresh period option does not touch it; the threshold and the two-minute interval are constants. Turn auto refresh off. What a member needed from it was learning servers, which it gets from the catalog, from attaching and from closer-peer answers, and dropping dead ones, which setRouterCatalog now does from the control plane's list, logged. Bootstrap still fills the table once at start. Connections to non-attached routers now happen only when the member looks something up or re-announces a service, for at most the auth deadline each.
How many routers hold a record, how often a member re-announces one and how long a router keeps it are properties of a mesh: a record a member stored on three routers is one another member looks for on three, and a record that expires between two announcements is absent most of the time. Each member and router setting them for itself is a way to get them wrong. Publish them from the control plane. ControlPlaneInfoResponse gains directory (DirectoryParameters: replication, reprovide_interval, and record_ttl, three intervals, derived), set by --dht-replication (3) and --dht-reprovide-interval (5m) on agentmesh-control-plane and agentmesh-one. The api package names the defaults and reads the message back into the settings a member or router applies, with the default for anything a control plane left unset, so a member started against an older control plane, or none, runs as a member of a default mesh. The consumers follow in their own commits.
The node re-announced every 5 minutes, a constant, and node and router each kept records for a TTL of their own, --dht-provider-addr-ttl, 15 minutes by default on both. The pair only works while the operator keeps them consistent on three binaries, and a TTL that falls under the interval makes a service absent most of the time. Both now come from the directory parameters the control plane publishes on /info. The node reads them with the rest of /info before it builds its DHT and at every sync: the reprovide loop takes the interval at each cycle, the record TTL applies when the DHT is built. The router reads /info once after its keys sync, before its DHT exists, and does not start without it, as it does not without the keys. Both log the parameters they run on. Gone with that: --dht-provider-addr-ttl and --dht-max-record-age on the node and the router, --router-dht-provider-addr-ttl and --router-dht-max-record-age on agentmesh-one, and the validation that checked the pair on the node. The value store the max record age bounded is not used by the mesh.
A member's DHT client opens connections to routers the member is not attached to, for a lookup or an announcement, to the routers kademlia puts closest to the key. Those connections were anonymous: the router closed them at its auth deadline a minute later and the next lookup opened them again, each time with a TLS handshake, identify on both sides and a GossipSub peer. Run the credential handshake on any outbound connection to a router the catalog lists, as the transit path already does for a circuit. The router then keeps the connection as a session and counts it, and the member's lookups reuse it. A member holds its K attached routers plus the routers its keys land on; more routers spread the second set, and the member's own connection manager trims the idle ones first. Two callers can now reach for the same router as its connection comes up, the attach path and this one, and a repeated handshake trips the router's per-peer limiter, so authRouter runs one handshake per router at a time and a second caller waits for it. agentmesh_node_routers gains state="directory"; agentmesh_node_directory_handshakes_total counts the handshakes by result.
Kademlia stores a record on the k servers closest to its key and asks those k for it; k was the library's 20 on members and routers alike. With fewer than 20 routers that is every router, for every lookup and every announcement, from every member, which is what the connection counts in google#277 measured, and adding routers did not change it. Apply the replication factor the control plane publishes (three by default) as k, and as the lookup's concurrency and resiliency, on the member: a record lives on the three routers closest to its key, a lookup asks those three, three at a time. The routers keep a bucket of 64 so their tables hold every router of a mesh and a FIND_NODE answer lists the closest routers to the key in order; the member keeps the first 2r of them, so it learns the r closest from one answer of a router it is attached to and reaches them in one hop. A member's sessions are then its K attached routers plus the r closest to each key it publishes or looks up, and a mesh with more routers spreads the second set instead of adding to it.
The guide's model counted a router's sessions as its attached members. With the directory sessions a member now keeps, a router holds about F × (K + D) / N: K attached routers per member (--routers, the member's own) and D directory routers, up to the mesh's replication per key the member publishes or looks up, never more than N − K. On few routers D is N − K and every member holds every router; on many it is a handful whatever N is, and adding routers divides both terms. State the model, what each of the three knobs buys and costs and where each is set, the hot key a service type is, and the sizing rule with examples that follow from it: six routers at 6000 hold about 4000 members that use the directory with one router out; 10,000 take twelve. The directory state of agentmesh_node_routers is what to watch for D, and the router's unauthenticated-close counter now reads strangers and members on an older release.
A JS member ran its kad-dht client with the library's k of 20: every lookup asked every router of a mesh with fewer than that. The control plane publishes the mesh's replication on /info; the member reads it there with the routers and the ban set (AgentMesh.dhtReplication), keeps the last value it saw when a pull publishes none, and joins with it as the client's kBucketSize and alpha, so a record it stores on the r routers closest to a key is one a Go member looks for on those r, and the other way round. DEFAULT_DHT_REPLICATION is the api default for a control plane that predates the field.
A Python member's DHT walk asked its routers and, when none had the record, followed every closer router the answers named, up to eight a round. A router now names the routers closest to the key in order, as many as it knows, and a record lives on the mesh's replication of them, so following more than that is dialing routers that cannot hold it. Read the replication from /info with the routers and the ban set (Mesh.dht_replication), keep the last value when a pull publishes none, and follow that many of the closer routers from one round to the next in both lookups. DEFAULT_DHT_REPLICATION is the api default for a control plane that predates the field.
69de3f7 to
78c8c4f
Compare
No description provided.