Skip to content

deploy: the cluster issuer is a workload issuer; --node-retention per environment - #729

Merged
aojea merged 1 commit into
google:mainfrom
aojea:testnet-workload-issuer
Oct 15, 2026
Merged

aojea merged 1 commit into
google:mainfrom
aojea:testnet-workload-issuer

Conversation

@aojea

@aojea aojea commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Found while auditing the bananas database after #701.

The testnets list the GKE service account issuer under --issuer, next to the human one (auth.sam-mesh.dev). A service account token therefore enrolls as a human: a 90-day session, and the token is accepted at /user/*. The probe and canary cronjobs enroll a fresh identity on every run, so bananas holds 19,601 admitted rows with sessions of ~2160h, none of them a member any more, and the node GC cannot touch an admitted row.

  • The GKE issuer moves to --workload-issuer, which the Kubernetes guide already prescribes: 48h sessions (--workload-session-ttl) renewed with a fresh projected token, refused at /user/* and /oauth/authorize. Nothing in-cluster calls /user/* with a service account token; the probes and canaries enroll through agentmesh-node and the SDKs.
  • --node-retention is rendered from a NODE_RETENTION environment variable (unset is the binary's 720h), so an environment that runs scale tests can set it to days and observe the sweep within the test window. I will set NODE_RETENTION=72h on bananas; hub keeps the default.

Verification plan: after rollout, the probe cronjobs keep passing, agentmesh_control_plane_enrolled_nodes{state="admitted"} on bananas stops growing with each probe run, and once their 48h sessions lapse plus the retention, agentmesh_control_plane_nodes_deleted_total{reason="session"} advances and agentmesh_control_plane_storage_rows{table="nodes"} (#728) shrinks.

… environment

The testnets listed the GKE service account issuer under --issuer, next to
the human one. A service account token therefore enrolled as a human: a
90-day session, and the token accepted at /user/*. The probe and canary
cronjobs enroll a fresh identity on every run, so bananas held 19,601
admitted rows with sessions of ~2160h, none of them a member any more.
Under --workload-issuer, which the Kubernetes guide already prescribes,
these are 48h sessions renewed with a fresh projected token, and the
control plane's node GC collects the rows of the runs that ended.

--node-retention is rendered from the NODE_RETENTION environment variable
(unset is the binary's 720h), so an environment that runs scale tests can
set it to days and watch the sweep within the test window.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Kubernetes template for the agentmesh control plane by splitting the --issuer flag into separate --issuer and --workload-issuer arguments, and introducing a new --node-retention argument. Feedback was provided to use shell default parameter expansion for ${NODE_RETENTION} (e.g., ${NODE_RETENTION:-720h}) to prevent the container from crashing on startup if the environment variable is unset or empty.

Comment thread .github/k8s/agentmesh-control-plane-template.yaml
@aojea
aojea merged commit ba033eb into google:main Oct 15, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant