Skip to content

feat(apexkube-agent): access + metrics agents in one release, separate tunnels - #157

Merged
Luv7804 merged 1 commit into
mainfrom
agent/feat/apexkube-agent-profiles
Oct 31, 2026
Merged

Luv7804 merged 1 commit into
mainfrom
agent/feat/apexkube-agent-profiles

Conversation

@Luv7804

@Luv7804 Luv7804 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What

One release can now run one or two agents from the same image:

access (default) metrics (metrics.enabled)
Downstream TLS plain HTTP
Client auth JWT verified none
Upstream kubernetes.default.svc:443 + TLS metrics.envoy.backend (thanos-receive:10908), plain HTTP
Service account token injected as Authorization: Bearer not injected, not mounted
Cluster-wide RBAC ClusterRole + binding none
Tunnel own wg0 + key own wg0 + key
Resources envoy.resources metrics.resources (bigger, for scrape bursts)
access:
  enabled: true      # unchanged defaults
metrics:
  enabled: false     # true adds a second Deployment and a second tunnel
  • Default install renders only the access agent, byte-identical to 2.0.0.
  • metrics.enabled: true adds <fullname>-metrics: its own Deployment, ServiceAccount, envoy ConfigMap, WireGuard ConfigMap and private-key Secret. Because the tunnel lives in the pod, scrapes consume only the metrics pod's CPU/memory limits and never share the access tunnel's queues.
  • access.enabled: false gives a metrics-only install, with no access pod, no certificate Secret and no cluster-wide RBAC at all.
  • The chart fails to render if both are disabled.

The access agent also gains envoy.backend.host/port, envoy.backend.tls.enabled, envoy.tls.enabled, envoy.jwt.enabled and envoy.injectServiceAccountToken, so a single-pod install can point elsewhere without deploying the second pod.

Requirement: the metrics agent needs its own WireGuard peer (own key pair and tunnel address), ideally with AllowedIPs scoped server-side to the Thanos endpoint.

Service account token fix

The Lua filter cached the projected token for the life of the worker, so a rotated token was never picked up (masked on k3s, which extends token expiry). It now reads the file per request — no clock dependency, since os.time() availability in Envoy's sandbox is not guaranteed.

Verification

  • Compatibility gate: helm template of 2.0.0 vs 2.1.0 for defaults, ci/ci-values.yaml and a real production values file — identical apart from the Lua block and the version-derived labels/checksums.
  • 52 render assertions: default profile, metrics-only, both together (two Deployments, distinct selectors, no shared names, access doc unchanged), config-override merge, disabled-both guard, required-private-key guard, and each access toggle independently.
  • helm lint clean for defaults, both CI files and values-metrics.yaml.
  • Live, one release with both agents on k3s: two pods, two distinct wg0 addresses (10.90.0.2/32 and 10.90.1.2/32), each pod returned 200 through its own envoy to a plain-HTTP echo backend; the access pod injected Authorization: Bearer <token>, the metrics pod injected nothing. Exactly one ClusterRole/ClusterRoleBinding existed for the release, both belonging to the access agent.
  • CI: ct install covers both profiles (ci-values.yaml and ci-metrics-values.yaml), so envoy is booted with each rendered config in a kind cluster.
  • Local release upgraded 2.0.0 → 2.1.0 earlier with real values: 0 restarts, /healthz 200 through the tunnel, /version 401 (JWT filter active).

Not in this PR

  • Rolling the fleet (monitoring, hydra11, hydra31, management-dev) — INTINF-428 / AK-196.
  • Deploying Thanos: no cluster has a Thanos service yet, so the metrics agent ships wired but unexercised against real Thanos.
  • inline_code in the Lua filter is deprecated by Envoy; switching to source_code.inline_string would change the default rendering, so it is left for a separate change.

Run the access agent (Kubernetes API; TLS, client JWT, service account token) and
optionally a second metrics agent in the same release. Both use the same image, but
each has its own Deployment, ServiceAccount, envoy config and WireGuard identity, so
scrape volume stays off the access pod and its tunnel.

Defaults deploy only the access agent and render identically to 2.0.0. Setting
metrics.enabled adds a plain-HTTP agent pointing at metrics.envoy.backend
(thanos-receive:10908 by default), with its own resources, no service account token
and no cluster-wide RBAC. access.enabled=false gives a metrics-only install.

The access agent also gains envoy.backend.host/port, envoy.backend.tls.enabled,
envoy.tls.enabled, envoy.jwt.enabled and envoy.injectServiceAccountToken. Its Lua
filter now reads the projected token per request instead of caching it, so a rotated
token is used on the next request and the filter no longer depends on os.time().
@Luv7804
Luv7804 force-pushed the agent/feat/apexkube-agent-profiles branch from 3696f3a to 375f84c Compare October 30, 2026 12:55
@Luv7804 Luv7804 changed the title feat(apexkube-agent): access and metrics profiles; fix cached SA token feat(apexkube-agent): access + metrics agents in one release, separate tunnels Oct 30, 2026
@Luv7804
Luv7804 merged commit eb90aa8 into main Oct 31, 2026
4 checks passed
@Luv7804
Luv7804 deleted the agent/feat/apexkube-agent-profiles branch October 31, 2026 03:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants