Repository navigation
Members ask their routers for the directory and listen as gossip leaves - #704
Merged
Merged
Conversation
A member asks a router it is attached to for the directory over /mesh/directory/1.0.0: one DirectoryRequest per stream, provide, find_providers or find_peer, one DirectoryResponse back. The member keeps no DHT of its own; the router runs the lookup across the routers' DHT. Routers place a member's record on each other with the same message, provider set, which only a router may do. A key is the sha256 multihash of a name, which is how go-libp2p-kad-dht keys provider records: "mesh:service:<type>[:<name>]" for a service, as today, and "mesh:peer:<peer_id>" for the member itself, so reaching a peer and reaching a service are the same lookup.
A member asks the router it is attached to, over one stream per request behind the auth gate. provide stores the member's record and places it on the routers closest to the key, as many as the mesh's replication, over the same protocol with the provider named; only an authenticated router may name one, so a member can only ever announce itself, at addresses that are its own. find_providers runs the lookup across the routers' DHT, which the routers keep among themselves. find_peer answers from this router's own connection for a member attached here and from the member's record under its peer key otherwise. Requests, results and placements are counted by operation and result, with a latency histogram; a malformed request is logged as the peer's doing, a failed lookup as the mesh's.
A member's kad-dht client is gone. The node announces what it serves and where it is reached to a router it is attached to, over /mesh/directory/1.0.0, and asks there for providers and peers; the routers answer from their own DHT. A request goes to one attached router and to the next when it fails, so one router restarting costs a request a timeout. Requests are counted by operation and result with a latency histogram. What a member no longer does: bootstrap a routing table, hear of every router and connect to the ones a key lands on, hold sessions with them, refresh or prune a table. Its sessions are with the K routers it is attached to and nothing else, so a router holds F × K / N sessions for a fleet of F members on N routers, and a join costs K handshakes. The service keys are the same sha256 multihashes as before, now from api.ServiceKey, so records written by a node that predates this are found. get_mesh_info and the mobile app report the attached routers where they reported a routing table size.
A member speaks floodsub, not gossipsub. A router delivers every control plane event to a floodsub peer outright, so the member hears bans, key rotations and policy updates from the routers it holds and takes no part in the routers' gossipsub mesh: no heartbeat, graft, prune or peer exchange per member on the router. The interest-scoped service announcements members gossiped among themselves (/mesh/discovery/v1, every ten seconds while a topic had subscribers) are gone with their consumers: the find_remote_tools fast path and the OpenAI facade's load hints, both of which fell back to the directory already. Discovery is the directory alone.
The kademlia walk is gone. A lookup is one DirectoryRequest to a router the member holds, over /mesh/directory/1.0.0, and to the next held router when that one fails; the router runs the lookup across the routers' DHT. The member connects to no router it does not hold and runs no handshake beyond those, and the replication the control plane publishes no longer concerns it. The member speaks floodsub instead of gossipsub: a router delivers every control plane event to it outright, and it takes no part in the routers' gossipsub mesh.
The kad-dht client is gone, and @libp2p/kad-dht with it. A lookup is one DirectoryRequest to a router the member holds, over /mesh/directory/1.0.0, and to the next held router when that one fails; the router runs the lookup across the routers' DHT. The member connects to no router it does not hold, runs no handshake beyond those, and the replication the control plane publishes no longer concerns it. The member speaks floodsub only: the gossipsub implementation is told to register that protocol alone, so a router delivers every control plane event to it outright and it takes no part in the routers' gossipsub mesh.
The sizing model: a fleet of F members on N routers, each holding K of them, puts F × K / N sessions on each router, and the directory costs a router no sessions, records and requests only. The member cost, the roles, the protocol table, the bounds, the sizing of the routers and the metrics follow. The router reference gains the directory step and its metrics; the node reference, node API and connecting guide report the attached routers where they reported a routing table; the concept pages describe the directory as the routers serve it and the gossip as members take it, as leaves.
kad-dht decides whether a peer enters its table when it handles that peer's identify event, on its own schedule, and drops a peer its filter refused. A router that authenticated between that decision and the drop stayed out of the table until it authenticated again, which a router that stays admitted never does. The federation pass now admits every authenticated peer router to the table on every tick; adding a peer the table holds is a no-op.
The egress assignments the control plane hands a node are registered before the node holds a router, so their first announcement had nobody to go to and the next was a reprovide interval away. Every path that attaches a router now runs a reprovide cycle, which the trigger coalesces: a router gained at start, by the top-up, or by a redial takes the node's records at once.
Routers that start at the same moment lease after each other's first federation pass, so each read a list without the other and the next pass was thirty seconds away; until then a lookup on one router could not reach a record on the other. The members' DHT clients used to bridge that gap by accident, and no longer connect to routers they do not hold. The federation loop runs every two seconds while the control plane lists a router this one has not authenticated with, and every thirty once it has them all. A directory request authenticates with every listed router not yet reached before it runs, and a lookup that found nothing re-reads the list, when the last read is older than a second, and tries once more.
A member announced its services to the directory but never itself, so a find_peer answered only through a router that held the peer, from that router's peerstore; through any other router the peer key had no record and the lookup was not found. Every announcement pass now ends with the node's record under api.PeerKey, after its services, so once the peer record names an address the service records of the pass do too. The directory client remembers the addresses of the record it last announced and /debug/network-info shows them as directory_addresses, next to the addresses the node announces now: what the directory holds for the node follows its addresses at the next pass, 2 s after they change, and a router that holds the node answers a lookup from what it sees of it, which is ahead of the records. TestCallerTransitsTheRouterItShares waited on that lookup and read the service record for the call, which named only the first router when the pass had not run yet; it waits on the records the provider announced. TestCallerTransitsARouterItsCatalogMissed pins the peer record: the caller's router does not hold the provider, so only the record answers.
The integration tests run a DNS server for the dnsaddr addresses routers announce and point agentmesh-node at it with AGENTMESH_TEST_DNS_SERVER; the router ignored it, so in TestRouterFederationAndRelay the two routers never resolved each other and never federated. Each member's record was on the one router it announced to, and a lookup through the other found nothing until a later pass happened to land there: the test passed on its retry loop, which itself ran the search once (an exec.Cmd started twice fails) and timed out whenever the one search was empty. The router now honours the variable as the node does. The test runs a fresh search per attempt and prints both routers' logs when it fails.
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Members no longer gossip ServiceAnnounce on /mesh/discovery/v1 topics: the directory is the one place a service is found. The message, the topic builder and its validation had no caller left, which the dead-code check in lint reports. Bindings regenerated.
The SDK README still described service discovery as a Kademlia lookup on /mesh/kad/1.0.0 with gossip announcements beside it. It now describes the directory request to a router the session holds, and names the files that derive the keys and serve the requests. The service-key fixture says the same.
get_mesh_info no longer reports dht_size; the check that the local node is on the mesh reads attached_routers instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A member no longer keeps a DHT or takes part in the routers' gossip mesh. It holds K routers and uses them for everything: it announces its services and where it is reached to one of them, asks one of them for providers and peers, and receives the control plane's events from them as a floodsub leaf. Sessions per router become F×K/N (F members, K routers held, N routers) instead of growing with every router a member's lookups touched, which is what ladder 14 measured (every member holding every router, 5.4–5.9k sessions per router at 5.8k members).
Protocol
/mesh/directory/1.0.0, oneDirectoryRequestper stream answered with oneDirectoryResponse(api/agentmesh.proto):provide(a record: key + addresses),find_providers,find_peer. Keys are sha256 multihashes ofmesh:service:<type>[:<name>]andmesh:peer:<peer_id>(api/directory_keys.go). The router keeps the member's record, places it on the routers closest to the key (the mesh's replication, from the control plane) by sending the same message withproviderset — a field only an authenticated peer router may set — and runs lookups across the routers' own kad-dht, which members no longer speak. The handler is registered through the auth gate from #700, so nothing is served before the handshake.Members speak floodsub (
/floodsub/1.0.0); the routers' gossipsub deliver every event to them and they take no part in heartbeats, grafts or peer exchange. Gossip-based discovery announcements (/mesh/discovery/v1) and the node's gossip fast path are gone; the directory is the one source.Changes
DirectoryProtocolID; generated bindings for Go, JS, Python.internal/router/directory.gohandler and placement; metricsagentmesh_router_directory_requests_total{op,result},_request_seconds{op},_placements_total{result}; federation pass runs at 2 s while listed routers are not all authenticated, and a directory request federates first (ensureFederated) and re-reads the router list once when a lookup finds nothing; an already-authenticated peer router is admitted to the table on every pass; the router honoursAGENTMESH_TEST_DNS_SERVERas the node does (the integration tests' routers never federated without it).Directoryclient over the held routers (internal/node/directory_client.go, metricsagentmesh_node_directory_requests_total{op,result},_request_seconds{op}); floodsub leaf; every announcement pass ends with the node's own record under its peer key so a lookup through a router that does not hold it finds it; services are announced again when a router is attached;/debug/network-infoshowsdirectory_addresses, the record last announced;dht_sizeandagentmesh_node_dht_routing_table_sizeare gone,attached_routersstays.@libp2p/kad-dhtremoved from package.json); directory client over the held routers; floodsub.TestCallerTransitsTheRouterItShareswaits on the records the provider announced;TestCallerTransitsARouterItsCatalogMissedpinsfind_peerthrough a router that does not hold the peer;TestRouterFederationAndRelayruns a fresh search per attempt and prints router logs on failure).Validation
make,make lint(0 issues),hack/verify-generated.sh,go test ./...(integration suite 6 consecutive clean runs), JS 130 and Python 167 SDK tests, SDK conformance integration tests with both SDKs built,make e2e-test31/32 (the failing Device Authorization Flow case is a/etc/agentmesh/node-config.yaml: permission deniedin the container that reproduces identically on main).Follows #700. Refs #277.