fix(controlplane, console): stop listing expired nodes on every refresh

[kaisoz]

The console's 15 s refresh downloaded every node row the control plane keeps for its retention period. On a mesh with probe CronJobs most of those rows are expired identities, so the console got slower every day.

• GET /admin/status and GET /user/status list admitted and banned nodes only. A user's list is filtered by owner in SQL, not in Go after loading the whole mesh.
• storage.Store lists nodes through one ListNodes(ctx, NodeFilter). The filter selects by state (admitted, expired, banned), owner and role, and pages with a keyset cursor and a limit. It replaces ListUnexpiredNodes and ListExpiredNodes.
• GET /user/nodes returns NodeListResponse: one page of nodes, newest enrollment first, with an opaque keyset page_token. state (repeatable) and role narrow the list. An administrator sees every owner's nodes and may pick one with owner_id. A user sees only their own nodes, and naming another owner returns 403. page_size defaults to 50, at most 200.
• An expired OIDC enrollment can renew in place with a fresh JWT, so it must stay revocable. The console's Nodes view gets a collapsed Expired Nodes section in internal/console/public/expired-nodes.js. It fetches state=expired only when opened, appends with Load more, removes a revoked row in place, and is cleared when the session ends.
• /metrics counts nodes by role and state with one COUNT(*) ... GROUP BY. It no longer loads every row, biscuits included, every 30 s.
• Migration 19 adds the indexes idx_nodes_owner_id (owner_id) and idx_nodes_enrolled_at (enrolled_at, peer_id).
• controlplane.Options.Default() sets NodeRetention to 30 days, so agentmesh-one deletes lapsed node rows too. A negative retention keeps them forever. --node-retention 0 still keeps them forever.

Timings

Measured locally with 25,040 node rows, 40 of them live, and 2 KB biscuits.

One console refresh, measured on the first revision of this PR. The status lists return the same rows in this revision.

	Before	After
Response size	54.6 MB	94 KB
Control-plane time	about 650 ms	33 ms
Browser refresh	1,359 ms	15 ms

One GET /user/nodes?state=expired page of 51 rows, query time:

Caller	Postgres 16	Postgres 16 after ANALYZE	SQLite	SQLite after PRAGMA optimize
Administrator	6.4 ms	0.03 ms	108 ms	0.17 ms
User who owns 3 of the rows	0.64 ms	0.02 ms	37 ms	0.07 ms

Postgres collects table statistics on its own through autovacuum, so its planner uses the new indexes. SQLite collects none unless asked. On SQLite, the default store of agentmesh-one, both new indexes stay unused and a page costs tens of milliseconds at this table size. Only opening the Expired Nodes section or pressing Load more pays that cost. Running PRAGMA optimize on SQLite is left for a follow-up.

Tests

Tests cover each state filter, the owner and role filters, keyset paging across equal enrollment times, a cursor from another owner's row, CountNodes against the same states, owner_id staying admin-only, claims staying admin-only, and the default retention deleting a row that lapsed more than 30 days ago. Two Playwright cases cover the console section. TestListNodes also passes against Postgres 16.

Follow-up commits (aojea)

Rebased on main and extended after the ladder runs on bananas showed the control plane stalling on a 119k-row nodes table.

• --node-retention must be positive. Options.Default fills an unset retention with 30 days as it does every other duration; Options.Validate and the flag refuse a value that is not positive. There is no "keep forever": an unbounded table is what this PR exists to prevent. agentmesh-one gets --control-plane-node-retention.
• ListNodes reads no blobs. public_key and biscuit_token (2 KB a row) stay out of every list; GetNode reads them for the one node a caller acts on. The admitted arm of nodeStateWhere is parenthesised like the others, and parseNodeStates drops a state named twice.
• A user's status lists admitted nodes only: banned rows are never collected and only an administrator can restore one. The console makes one status call per refresh (an admin's later refreshes read /admin/status alone).
• A store that does not answer is 503, not 401. admittedNode tells a store failure from a missing credential; the mesh endpoints answer 503 with one log line, the OAuth paths temporarily_unavailable. A database stall on the testnet had produced 484 silent 401s in one step of the ladder.
• The GC sweeps at a tenth of the retention, within a second and an hour, so the default still sweeps hourly and a short retention is honoured within seconds.
• Integration test TestLapsedMembersAreCollected: the testnet in seconds. Sixteen members enroll and leave, one refreshes with a fresh token; the lapsed ones are refused at /refresh, out of the status, paged by GET /user/nodes?state=expired, counted by /metrics, and deleted once the retention passes while the live one is untouched.

Not in this PR: a bootstrap enrollment still has no session, so a node that joined with a token and left is never collected. That is a separate change with its own trade-off (a member offline longer than the session must re-enroll) and follows in its own PR.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[aojea]

should we garbage collect or add an option to allow admins to define a time to purge expired nodes

[aojea]

Talked offline, I think we should find a way to improve the list of nodes with multiple discriminators for filtering and paging instead of adding a new API for each Node state ... that will scale better

[aojea]

@kaisoz

• Consolidate node listing into ListNodes(ctx, NodeFilter) in storage.Store: Replace ListUnexpiredNodes and ListExpiredNodes with a NodeFilter struct on ListNodes that supports filtering by state (Admitted, Expired, Banned), OwnerID, Role, and keyset pagination (Cursor, Limit). Use that same filter in HandleUserStatus so non-admin owner_id filtering happens in SQL rather than in Go after loading all mesh nodes.
• Generalize the HTTP & Proto API (GET /user/nodes + NodeListResponse): Instead of GET /user/expired-nodes and ExpiredNodesResponse, introduce NodeListResponse (matching BootstrapTokenListResponse / EnrollmentRequestListResponse) and GET /user/nodes?state=expired&page_size=...&page_token=... so the same endpoint can page through admitted, banned, or expired nodes (and filter by owner_id / role).
• Fix /metrics full-table scan in readMeshSnapshot (internal/controlplane/metrics.go:297): readMeshSnapshot still calls store.ListNodes(ctx) every 30s, loading all 25k+ rows (including biscuit_token blobs) into memory just to count nodes by (role, state). Replace this with a SQL COUNT(*) ... GROUP BY method on Store.
• Add an index covering owner_id and default NodeRetention in Options.Default(): Add an index for owner_id on nodes so per-owner queries don't scan all workload rows, and set NodeRetention = DefaultNodeRetention in controlplane.Options.Default() so agentmesh-one also garbage-collects expired nodes by default.

[aojea]

good directiion but instead of fixing the symptom let's fix the root cause

[kaisoz]

good directiion but instead of fixing the symptom let's fix the root cause

Done. Also added a table with timings to the description

[aojea]

Overall the ListNodes(ctx, NodeFilter) consolidation, GET /user/nodes keyset pagination, and CountNodes metrics query look great. Left a few inline comments on NodeRetention semantics, SQL expression parenthesization, state parameter deduplication, and avoiding unnecessary payloads in status polls.