Repository navigation
controlplane: bootstrap members that fall silent are forgotten; ephemeral tokens; every table enrollment grows is bounded - #728
Merged
Conversation
…eral tokens A bootstrap enrollment has no session, so the node GC, which deletes rows whose session lapsed, never matched one: every member that joined with a token and left stayed enrolled for good. The ladder tests alone left a hundred thousand such rows on the testnet, and the control plane's full scans of that table were what stalled it under load. The GC now also deletes an unbanned bootstrap row that has not refreshed for --node-retention: enrolled_at is rewritten by every refresh, so it already is the last time the member was heard from. A member that keeps refreshing keeps its row; one that fell silent for a month is forgotten, and enrolls again if it comes back, as a self-hosted CI runner is removed after a fortnight without connecting and re-registers. Nothing a running member can observe changes: no session, no refusal, no new state. A bootstrap token can be ephemeral (BootstrapTokenCreateRequest.ephemeral, a checkbox in the console). The members it enrolls carry the flag and are forgotten after --ephemeral-node-retention (an hour) without a refresh: a fleet in a scale test, a probe, a CI job. The ladder and journey harnesses mint ephemeral tokens, so a scale run leaves the table as it found it within the hour. agentmesh_control_plane_nodes_deleted_total gains a reason label: session, silent, ephemeral. Migration 20 adds the ephemeral column to tokens and nodes. agentmesh-one exposes the retention as --control-plane-ephemeral-node-retention.
TestSilentEphemeralMembersAreForgotten is the ladder's fleet in seconds: twelve members enroll with an ephemeral token and fall silent, two with a regular one, of which one refreshes. The fleet is gone after the ephemeral retention, counted under reason=ephemeral; the regular members stay within their retention and refresh; a forgotten member that comes back is a stranger and is told to enroll again. The scale harnesses mint their tokens ephemeral.
…catalog cache
The node GC left four other things growing with every enrollment on the
testnet: 99,990 enrollment requests (auto-approval writes one per member
and nothing deleted them), 290 router leases for 13 live routers, 325
bootstrap tokens for 9 live ones, and one in-memory service report per
member that ever reported, dropped only on a ban.
One sweep now covers all of them. Enrollment requests resolved or left
pending for --node-retention go, for --ephemeral-node-retention when an
ephemeral token made them; bootstrap tokens expired for the retention go
once no request refers to them (the request carries a foreign key to its
token, so requests go first); router leases expired for the retention go;
and the catalog cache keeps a report only for a node still admitted. Each
step logs its own failure and the next one still runs.
Two metrics make a reaper that stopped visible without a database shell:
agentmesh_control_plane_storage_rows{table} is the row count of every
table that grows with mesh activity, read with the mesh state snapshot,
and agentmesh_control_plane_storage_rows_deleted_total{table} counts what
each sweep removed. agentmesh_control_plane_catalog_entries is the size
of the in-memory cache.
Store round trips cover each delete and the count; the controlplane unit
test runs one sweep over all four and the cache; the ephemeral-fleet
integration test asserts the fleet's requests go with its rows while the
regular members' stay, and that the gauges are served.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces ephemeral bootstrap tokens and nodes, allowing nodes enrolled via these tokens to be garbage-collected after a shorter retention period (defaulting to one hour) once they fall silent. It adds the --ephemeral-node-retention configuration, updates database schemas and queries, expands garbage collection to cover stale enrollment requests, expired tokens, and router leases, and introduces new Prometheus metrics for storage monitoring. The review feedback highlights a performance concern in the SQL store where the use of COALESCE prevents index utilization on the enrollment_requests table, and identifies a style guide violation regarding auditability, recommending debug logging of peer IDs when pruning the service catalog.
…eport is traced Review: COALESCE(resolved_at, created_at) < ? cannot use the index on created_at; resolved_at < ? OR (resolved_at IS NULL AND created_at < ?) can. And a peer whose service report the sweep drops is a decision about that peer, so it leaves a debug line with its ID; the count stays at info.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #701. The node GC there matches a node whose session expired; a member enrolled with a bootstrap token has no session (
expires_atis zero), so the 10,000 rows each scale ladder left on bananas were never collected, and a control plane audit after the merge found four more tables and one in-memory cache growing with every enrollment.Bootstrap members are forgotten after silence (
--node-retention, 30d)Every
/refreshrewritesenrolled_at, so the row already records when the member was last seen. A bootstrap member that has not refreshed for--node-retentionis deleted; one that comes back after that enrolls again (its credential still verifies,/refreshanswers 401 "enroll again"). No new session semantics, no behaviour change for a member that refreshes.Ephemeral tokens (
--ephemeral-node-retention, 1h)A fleet, a CI job or a probe that enrolls a fresh identity on every run leaves one row per run.
BootstrapTokenCreateRequest.ephemeralmarks a token whose members are kept for--ephemeral-node-retentionafter their last refresh instead of a month. The flag is copied onto the node row at enrollment, shown in the console and the token/node views, and the scale scripts mint ephemeral tokens. This is the kubeadm model: the token says how long its members are remembered.One sweep bounds every table that grows with enrollment
The bananas audit (2026-10-15):
enrollment_requests99,990 rows (auto-approval writes one per member, nothing deleted them),routers290 for 13 live,bootstrap_tokens325 for 9 live, and one in-memory service report per member that ever reported, dropped only on a ban. The same hourly sweep now deletes requests resolved or pending for the retention (the ephemeral retention when an ephemeral token made them), tokens expired for the retention that no request refers to (requests go first; the FK), router leases expired for the retention, and the catalog entry of every peer that is no longer admitted. Each step logs its own failure and the next still runs.A reaper that stops is visible
agentmesh_control_plane_storage_rows{table}: row count of every growing table, read with the mesh state snapshotagentmesh_control_plane_storage_rows_deleted_total{table}andagentmesh_control_plane_nodes_deleted_total{reason=session|silent|ephemeral}agentmesh_control_plane_catalog_entriesRisk
Retention-gated, idempotent deletes; defaults (30d / 1h) remove nothing a running mesh needs. Both retentions must be positive (no "keep forever"). Schema migration 20 adds a
BOOLEAN DEFAULT FALSE NOT NULLcolumn tobootstrap_tokensandnodes. The only new hot-path query is a per-tableCOUNT(*)every 30s in the metrics snapshot.Tests
Store round trips for each delete and the count; unit tests for the silent/ephemeral sweep, the four-table sweep and the catalog prune, the ephemeral flag propagation, and the positive-retention validation; integration
TestSilentEphemeralMembersAreForgotten(12 ephemeral + 2 regular members,--ephemeral-node-retention 1s): the fleet is gone from/admin/statuswithin seconds, its requests go with its rows, the regular members refresh, a forgotten member gets 401, and the gauges are served. Build,make lint, unit, integration,verify-generated/verify-sdk-generated, console UI tests pass.Review
resolved_at < ? OR (resolved_at IS NULL AND created_at < ?)so the existingidx_enrollment_requests_created_atserves it;COALESCEover the two columns could not use an index.DeleteNodes(ctx, NodeFilter): the sweep is the only caller, each predicate is fixed per table (and three of the five are on tables without a filter type), thebanned = falseguard stays in the SQL instead of in every caller's filter, and a zero filter cannot mean "delete everything". A filtered delete fits an operator-facing bulk forget (DELETE /admin/nodes?state=expired), which can introduce it when it arrives, with the cutoff fields the list view would want too.Testnet
Bananas held 19,601 admitted OIDC rows from probe runs with ~90-day sessions (#729 moves the cluster issuer to
--workload-issuer). The finished runs, those not connected to any live router and silent for over an hour (19,307), hadexpires_atset to now rather than being deleted, so the deployed sweep collects them after the 72h retention andnodes_deleted_total{reason="session"}is observed on real data. Live members, routers, requests, tokens and leases were left for the reapers in this PR.