Skip to content

Member directory - #693

Merged
aojea merged 9 commits into
google:mainfrom
aojea:member-directory
Oct 13, 2026
Merged

aojea merged 9 commits into
google:mainfrom
aojea:member-directory

Conversation

@aojea

@aojea aojea commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces configurable DHT replication and reprovide intervals for nodes, allowing better control over how service records are published and looked up across routers. It also establishes authenticated directory sessions with unattached routers to reuse connections, and prunes the DHT routing table when routers leave the catalog. The review feedback suggests optimizing lock contention and reducing memory allocations in internal/node/directory.go by consolidating lock acquisitions and avoiding temporary map allocations in authDirectoryRouter and DirectoryRouters.

Comment on lines +34 to +37
func (n *AgentMeshNode) authDirectoryRouter(ctx context.Context, router peer.ID) {
if n.isAttached(router) || n.isAuthenticatedAndConnected(router) || !n.isCatalogRouter(router) {
return
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

6. Behavior must be auditable from logs and metrics / Efficiency

The authDirectoryRouter function currently performs multiple sequential lock acquisitions on n.mu by calling n.isAttached(router), n.isAuthenticatedAndConnected(router), and n.isCatalogRouter(router) back-to-back. Each of these helper functions acquires and releases the lock individually, leading to unnecessary lock contention and overhead.

We can optimize this by acquiring the lock once, extracting the required state, and performing the checks.

func (n *AgentMeshNode) authDirectoryRouter(ctx context.Context, router peer.ID) {
	n.mu.Lock()
	attached := n.attached[router]
	authenticated := n.authenticatedRouters[router]
	isCatalog := false
	for _, c := range n.routerCatalog {
		if c.ID == router {
			isCatalog = true
			break
		}
	}
	n.mu.Unlock()

	if attached || (authenticated && n.Host.Network().Connectedness(router) == network.Connected) || !isCatalog {
		return
	}
References
  1. The style guide emphasizes clean, robust, and maintainable Go code. Minimizing lock contention and avoiding unnecessary allocations in hot paths like connection callbacks aligns with these principles. (link)

Comment thread internal/node/directory.go Outdated
Comment on lines +68 to +82
func (n *AgentMeshNode) DirectoryRouters() int {
attached := map[peer.ID]bool{}
n.mu.Lock()
for r := range n.attached {
attached[r] = true
}
n.mu.Unlock()
count := 0
for _, r := range n.authenticatedRouterIDs() {
if !attached[r] {
count++
}
}
return count
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

6. Behavior must be auditable from logs and metrics / Efficiency

The DirectoryRouters function currently allocates a temporary map attached and calls n.authenticatedRouterIDs(), which locks n.mu again and allocates a temporary slice of peer IDs. This results in multiple lock acquisitions and unnecessary memory allocations.

We can optimize this by locking n.mu once and performing the checks directly on the live maps.

func (n *AgentMeshNode) DirectoryRouters() int {
	n.mu.Lock()
	defer n.mu.Unlock()
	count := 0
	for r := range n.authenticatedRouters {
		if !n.attached[r] && n.Host.Network().Connectedness(r) == network.Connected {
			count++
		}
	}
	return count
}
References
  1. The style guide emphasizes clean, robust, and maintainable Go code. Minimizing lock contention and avoiding unnecessary allocations in hot paths like connection callbacks aligns with these principles. (link)

aojea added 9 commits October 13, 2026 14:22
The node ran its DHT in ModeAuto, so a member AutoNAT found publicly
reachable became a DHT server: listed in every other member's routing
table, taking their lookups on a 400-connection budget, answering
whoever connected, and trusted for unsigned provider records. The
routers are the directory (the roles table says so, the JS SDK is
clientMode, the Python SDK has no server), and nothing in the mesh
needs a member to serve it.

Fix the mode to client. --reachability keeps deciding how a member is
reached, direct or through a relay, and nothing else. The DHT options
move to dhtOptions so the test can build a member DHT on a host with
forced public reachability and check it stays a client while a ModeAuto
DHT on the same setup turns server.
A member's kad-dht client kept a connection to every router it was not
attached to about four fifths of the time, and at 10,000 members each
router accepted 3,900 to 5,100 inbound connections a minute, 99.9% of
them closed at the auth deadline without a handshake (google#277). The driver
is kad-dht's table maintenance, not lookups: a table of ten peers or
fewer counts as starved, so fixLowPeers triggers a refresh every two
minutes, and every refresh pings each entry and runs a lookup for the
member's own ID against all of them. With six routers as the only
servers that is a dial to every other router, closed a minute later,
opened again a minute after that. The refresh period option does not
touch it; the threshold and the two-minute interval are constants.

Turn auto refresh off. What a member needed from it was learning
servers, which it gets from the catalog, from attaching and from
closer-peer answers, and dropping dead ones, which setRouterCatalog
now does from the control plane's list, logged. Bootstrap still fills
the table once at start. Connections to non-attached routers now
happen only when the member looks something up or re-announces a
service, for at most the auth deadline each.
How many routers hold a record, how often a member re-announces one and
how long a router keeps it are properties of a mesh: a record a member
stored on three routers is one another member looks for on three, and a
record that expires between two announcements is absent most of the
time. Each member and router setting them for itself is a way to get
them wrong.

Publish them from the control plane. ControlPlaneInfoResponse gains
directory (DirectoryParameters: replication, reprovide_interval, and
record_ttl, three intervals, derived), set by --dht-replication (3) and
--dht-reprovide-interval (5m) on agentmesh-control-plane and
agentmesh-one. The api package names the defaults and reads the message
back into the settings a member or router applies, with the default for
anything a control plane left unset, so a member started against an
older control plane, or none, runs as a member of a default mesh. The
consumers follow in their own commits.
The node re-announced every 5 minutes, a constant, and node and router
each kept records for a TTL of their own, --dht-provider-addr-ttl, 15
minutes by default on both. The pair only works while the operator
keeps them consistent on three binaries, and a TTL that falls under the
interval makes a service absent most of the time.

Both now come from the directory parameters the control plane publishes
on /info. The node reads them with the rest of /info before it builds
its DHT and at every sync: the reprovide loop takes the interval at
each cycle, the record TTL applies when the DHT is built. The router
reads /info once after its keys sync, before its DHT exists, and does
not start without it, as it does not without the keys. Both log the
parameters they run on.

Gone with that: --dht-provider-addr-ttl and --dht-max-record-age on the
node and the router, --router-dht-provider-addr-ttl and
--router-dht-max-record-age on agentmesh-one, and the validation that
checked the pair on the node. The value store the max record age
bounded is not used by the mesh.
A member's DHT client opens connections to routers the member is not
attached to, for a lookup or an announcement, to the routers kademlia
puts closest to the key. Those connections were anonymous: the router
closed them at its auth deadline a minute later and the next lookup
opened them again, each time with a TLS handshake, identify on both
sides and a GossipSub peer.

Run the credential handshake on any outbound connection to a router the
catalog lists, as the transit path already does for a circuit. The
router then keeps the connection as a session and counts it, and the
member's lookups reuse it. A member holds its K attached routers plus
the routers its keys land on; more routers spread the second set, and
the member's own connection manager trims the idle ones first.

Two callers can now reach for the same router as its connection comes
up, the attach path and this one, and a repeated handshake trips the
router's per-peer limiter, so authRouter runs one handshake per router
at a time and a second caller waits for it. agentmesh_node_routers
gains state="directory"; agentmesh_node_directory_handshakes_total
counts the handshakes by result.
Kademlia stores a record on the k servers closest to its key and asks
those k for it; k was the library's 20 on members and routers alike.
With fewer than 20 routers that is every router, for every lookup and
every announcement, from every member, which is what the connection
counts in google#277 measured, and adding routers did not change it.

Apply the replication factor the control plane publishes (three by
default) as k, and as the lookup's concurrency and resiliency, on the
member: a record lives on the three routers closest to its key, a
lookup asks those three, three at a time. The routers keep a bucket of
64 so their tables hold every router of a mesh and a FIND_NODE answer
lists the closest routers to the key in order; the member keeps the
first 2r of them, so it learns the r closest from one answer of a
router it is attached to and reaches them in one hop. A member's
sessions are then its K attached routers plus the r closest to each key
it publishes or looks up, and a mesh with more routers spreads the
second set instead of adding to it.
The guide's model counted a router's sessions as its attached members.
With the directory sessions a member now keeps, a router holds about
F × (K + D) / N: K attached routers per member (--routers, the member's
own) and D directory routers, up to the mesh's replication per key the
member publishes or looks up, never more than N − K. On few routers D
is N − K and every member holds every router; on many it is a handful
whatever N is, and adding routers divides both terms. State the model,
what each of the three knobs buys and costs and where each is set, the
hot key a service type is, and the sizing rule with examples that
follow from it: six routers at 6000 hold about 4000 members that use
the directory with one router out; 10,000 take twelve.

The directory state of agentmesh_node_routers is what to watch for D,
and the router's unauthenticated-close counter now reads strangers and
members on an older release.
A JS member ran its kad-dht client with the library's k of 20: every
lookup asked every router of a mesh with fewer than that. The control
plane publishes the mesh's replication on /info; the member reads it
there with the routers and the ban set (AgentMesh.dhtReplication), keeps
the last value it saw when a pull publishes none, and joins with it as
the client's kBucketSize and alpha, so a record it stores on the r
routers closest to a key is one a Go member looks for on those r, and
the other way round. DEFAULT_DHT_REPLICATION is the api default for a
control plane that predates the field.
A Python member's DHT walk asked its routers and, when none had the
record, followed every closer router the answers named, up to eight a
round. A router now names the routers closest to the key in order, as
many as it knows, and a record lives on the mesh's replication of them,
so following more than that is dialing routers that cannot hold it.

Read the replication from /info with the routers and the ban set
(Mesh.dht_replication), keep the last value when a pull publishes
none, and follow that many of the closer routers from one round to the
next in both lookups. DEFAULT_DHT_REPLICATION is the api default for a
control plane that predates the field.
@aojea
aojea merged commit c443522 into google:main Oct 13, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant