feat(infra): show pod restarts and last termination reason

[Makisuo]

Problem

The pod pages show memory against limit, but not restarts or why a container last terminated. The only way to spot an OOM kill was a memory curve that drops after a peak.

What the receiver emits (pinned contrib v0.151.0)

• k8s.container.restarts: int gauge, the kubelet's RestartCount per container. Resource attrs include k8s.pod.name, k8s.namespace.name, k8s.container.name and the workload names. On by default.
• k8s.container.status.last_terminated_reason: a resource attribute on every k8s.container.* metric, set from LastTerminationState.Terminated.Reason (OOMKilled, Error, Completed, ...). Off upstream, but already enabled in values.yaml (presets.clusterMetrics.resourceAttributes), so no chart behaviour change was needed. I only added a comment so it doesn't get removed as unused.
• k8s.container.status.reason / k8s.container.status.state: optional sum metrics for the current waiting/terminated reason (for example CrashLoopBackOff). I left these off. Each one emits 9 or 3 series per container on every interval, and the last-terminated reason already answers "why did it restart".
• k8s_events (the k8sobjects receiver, off by default): each event lands as a log. The body is the whole Event object. The attributes are event.domain=k8s, event.name=<event object name> and k8s.resource.name=events, and the only resource attribute is k8s.namespace.name. The pod is only in body.object.regarding.name, so a join means parsing JSON bodies. Also, OOMKilling is a node-problem-detector event on the Node rather than the Pod. Not used here.

Changes

• Query podRestartsQuery in ch/queries/infra.ts. One row per (pod, namespace, container): restarts (last sample minus first in the window, floored at 0), totalRestarts (latest sample), and lastTerminatedReason (argMax by time). It filters by OrgId and window, with optional page pod names, namespace and workload. The MCP infra tools landed their own copy on main while this was open; the merge keeps this one, which takes the same opts and returns the same fields plus lastTerminatedReason, and adds that field to the MCP test fixtures. There are benchmark builder entries for the workload and page shapes.
• Endpoint POST /pod-restarts (PodRestartsRequest/PodRestartsResponse) plus a registry query and an API handler.
• Pod list: after the page loads, a second query fetches restarts for the visible pod names. Pods that restarted in the window get an inline marker beside the name, like 3 restarts · OOMKilled. It's red for OOMKilled and amber for other reasons. The full total is in the hover title.
• Pod page: a fifth "Restarts" tile shows the window count, the total, and "Last terminated: ". It reads "not reported" when the cluster collector has no rows for the pod. There's also a "Container restarts" table, shown once any container has restarted or terminated.

Caveat: kubelet can reset RestartCount when it prunes dead containers. Taking last minus first (rather than max minus min) means a reset inside the window can undercount but never overcounts.

Follow-up

Follow-up: the MCP inspect_infra pod view already calls podRestartsQuery and should now render lastTerminatedReason from podRestartsQuery, so an agent can name an OOM kill rather than infer it from memory.

Tests

• packages/query-engine: infra.test.ts (new podRestartsQuery cases) and benchmark/catalog.test.ts
• apps/web: components/infra/pod-restarts.test.ts (per-pod folding, reason choice, namespace separation, tone)
• Scoped typechecks: query-engine, domain, web, api

Not verified against a live cluster or in the browser.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Summary by CodeRabbit

• New Features
  • Pod lists and detail pages now show restart counts, including recent and total restarts, and the latest termination reason when available.
  • Pod restart details can be filtered by time range, namespace, or selected pod names.
  • Restart indicators distinguish OOM kills from other termination reasons.
• Bug Fixes
  • Restart counts now reflect changes between the earliest and latest readings in the selected time window, without showing negative counts.

[maple-review-bot]

Maple review

🟢 Confidence 7/10 · likely safe to merge
The new query's restarts aggregate is pinned by a test that asserts the wrong expression, so the one number the feature exists to show can be wrong.
_{quality 90/100 · 1 warning · tests partial · risk medium · 2/2 new units observable}

Adds a pod-restarts query, endpoint and UI: window restarts, the lifetime total and the last termination reason per pod. The restarts aggregate overcounts when the kubelet resets the counter inside the window; the open WarehouseInvalidInputError in warehouse-query-atoms.ts is untouched by this diff.

• podRestartsQuery aggregates per-container restarts, total and last termination reason
• POST /pod-restarts endpoint, registry query and web atom
• Pod list marks restarted pods inline; pod page adds a Restarts tile and container table

Findings

🟠 Warning · F1 · restarts overcounts when the kubelet resets the counter inside the window

correctness · packages/query-engine/src/ch/queries/infra.ts:706-707

max(Value) - min(Value) is the window's restarts only while the series is monotonic. The reset the comment on infra.ts:679 names as harmless puts samples like 50, 51, 0, 1 in the window, and this returns 51: the container's whole-life total, toned red in the pod list and labelled as this window's restarts on the tile. max(Value) - argMin(Value, TimeUnix) is the first-sample baseline form that actually holds the "never over" property the comment claims (the assertion at infra.test.ts:308 pins the old expression and needs updating).

			restarts: CH.ifNotFinite(CH.max_($.Value).sub(CH.argMin($.Value, $.TimeUnix)), 0),

🤖 Prompt to fix this finding with an AI agent

Findings from an automated review of commit f8a0e635293c5c960d3072d892bb6aea778e4d31. Verify each one against the current code before changing anything, fix only those that still apply, and keep each fix to the lines it names.

---

F1 · Warning · correctness · packages/query-engine/src/ch/queries/infra.ts:706-707
`restarts` overcounts when the kubelet resets the counter inside the window
`max(Value) - min(Value)` is the window's restarts only while the series is monotonic. The reset the comment on infra.ts:679 names as harmless puts samples like 50, 51, 0, 1 in the window, and this returns 51: the container's whole-life total, toned red in the pod list and labelled as this window's restarts on the tile. `max(Value) - argMin(Value, TimeUnix)` is the first-sample baseline form that actually holds the "never over" property the comment claims (the assertion at infra.test.ts:308 pins the old expression and needs updating).
Replace those lines with:
			restarts: CH.ifNotFinite(CH.max_($.Value).sub(CH.argMin($.Value, $.TimeUnix)), 0),

Production impact

Open errors in the changed files

Issue	Service	Occurrences	File
@maple/web/errors/WarehouseInvalidInputError	maple-web	12	apps/web/src/lib/services/atoms/warehouse-query-atoms.ts

After this merges, Maple checks whether they stop.

Production traffic of the changed files (last 7 days)

File	Calls/day	Busiest operations
apps/api/src/routes/internal/query-engine.http.ts	5	podInfraTimeseries 5/day, 0.0% err
packages/domain/src/http/query-engine.ts	5	podInfraTimeseries 5/day, 0.0% err

What was checked

• Org scoping kept: OrgId filter in the query (infra.ts:715) and CurrentTenant.Context in the handler (query-engine.http.ts:1556)
• Pod names go through CH.inList and an empty page drops the clause instead of emitting IN ()
• New endpoint rides the group's HttpMiddleware.tracer instrumentation (apps/api/src/http/api-observability.ts:28)

Observability coverage: 2 of 2 changes observable

Change	Kind	Observable	Evidence
POST /internal/query-engine/pod-restarts	server	yes	HttpApiBuilder group covered by ApiObservabilityLive's HttpMiddleware.tracer, apps/api/src/http/api-observability.ts:28-35
podRestartsQuery (ClickHouse)	client	yes	runs through the shared WarehouseQueryService.compiledQuery path like every registry query (query-engine.http.ts:1559)

_{f8a0e63 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 8c4119bc-0110-4ab6-9759-79b4f48e1634

---

📥 Commits

Reviewing files that changed from the base of the PR and between 2cb45d7 and cd60ce5.

---

📒 Files selected for processing (19)

• apps/ai/src/mcp/__evals__/infra-world.ts
• apps/ai/src/mcp/tools/__tests__/infra-tools.test.ts
• apps/api/src/routes/internal/query-engine.http.ts
• apps/web/src/api/warehouse/infra.ts
• apps/web/src/components/infra/kubernetes/pod-restarts-panel.tsx
• apps/web/src/components/infra/pod-restarts.test.ts
• apps/web/src/components/infra/pod-restarts.ts
• apps/web/src/components/infra/pod-table.tsx
• apps/web/src/lib/services/atoms/warehouse-query-atoms.ts
• apps/web/src/routes/infra/kubernetes/pods/$podName.tsx
• apps/web/src/routes/infra/kubernetes/pods/index.tsx
• deploy/k8s-infra/values.yaml
• packages/domain/src/http/query-engine.ts
• packages/query-engine/src/__sql_baseline__/catalog.sql
• packages/query-engine/src/benchmark/builders.ts
• packages/query-engine/src/ch/index.ts
• packages/query-engine/src/ch/queries/infra.test.ts
• packages/query-engine/src/ch/queries/infra.ts
• packages/query-engine/src/registry/queries.ts

---

 _________________________________________________________________
< You are pretty high on the list of people I want to help today. >
 -----------------------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

---

---

---

---

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[maple-review-bot]

1 inline note from Maple's review. The score and summary are in the review comment above.

[maple-review-bot]

Maple review

🟢 Confidence 7/10 · likely safe to merge
podRestartsQuery still reports a whole-life restart total when the kubelet resets the counter mid-window (open F1); the rest is contained and pinned by the new tests.
_{quality 90/100 · 1 warning · tests covered · risk medium · 2/2 new units observable}

Adds a podRestartsQuery, POST /pod-restarts, and pod-page/pod-list restarts UI (window count, total, last termination reason). Only the SQL baseline changed since the last review; the open overcount finding still stands, so this is not ready as is. The open WarehouseInvalidInputError in warehouse-query-atoms.ts is untouched by this diff.

• podRestartsQuery reads k8s.container.restarts per pod/namespace/container with argMax on the last-terminated reason
• New POST /pod-restarts endpoint, registry query and podRestarts web atom
• Pod list fetches restarts per page and marks rows inline; pod page adds a restarts tile and per-container table

Still open from earlier reviews

• 🟠 Warning · F1 · restarts overcounts when the kubelet resets the counter inside the window · packages/query-engine/src/ch/queries/infra.ts:706

Production impact

Open errors in the changed files

Issue	Service	Occurrences	File
@maple/web/errors/WarehouseInvalidInputError	maple-web	12	apps/web/src/lib/services/atoms/warehouse-query-atoms.ts

After this merges, Maple checks whether they stop.

Production traffic of the changed files (last 7 days)

File	Calls/day	Busiest operations
apps/api/src/routes/internal/query-engine.http.ts	5	podInfraTimeseries 5/day, 0.0% err
packages/domain/src/http/query-engine.ts	5	podInfraTimeseries 5/day, 0.0% err

What was checked

• podRestartsQuery filters OrgId and the window; baseline SQL at catalog.sql:1374 matches the builder fixtures
• The new POST /pod-restarts handler follows the neighbouring runQuery(Queries.*, tenant, payload) pattern and gets tenant from CurrentTenant.Context
• summarizePodRestarts keys rows by namespace/podName, covered by the namespace-separation test

Observability coverage: 2 of 2 changes observable

Change	Kind	Observable	Evidence
POST /pod-restarts (apps/api internal query-engine route)	Server	yes	HttpMiddleware.tracer stamps server spans for every HttpApi route (apps/api/src/http/api-observability.ts:6-24); handler adds no manual span, matching its neighbours
podRestartsQuery warehouse read	Client	yes	WarehouseQueryService.compiledQuery emits a span with context: "podRestarts" (packages/backend/src/services/warehouse/WarehouseQueryService.test.ts:1407-1411 asserts the span shape)

_{41f96a4 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}

[maple-review-bot]

Maple review

🟢 Confidence 8/10 · likely safe to merge
The restart-count fix (last minus first, floored) is pinned by the SQL test; the only gap is that the domain docs and the UI wording were not moved with it.
_{quality 98/100 · 1 note · tests covered · risk medium · 2/2 new units observable}

Adds a podRestartsQuery, a POST /pod-restarts endpoint and pod list/detail UI for container restarts and the last termination reason, and reworks the window count to last minus first so a kubelet counter reset can no longer overcount (F1 resolved). Safe to merge; the open WarehouseInvalidInputError issue is untouched by this diff.

• podRestartsQuery returns lastTerminatedReason and counts restarts as last − first, floored at 0
• New POST /pod-restarts endpoint wired through the registry and the web atom family
• Pod list marks restarting pods inline; pod page adds a Restarts tile and a container table

Findings

🔵 Note · F2 · PodRestartsResponse docs still say max − min while the query returns last − first

maintainability · packages/domain/src/http/query-engine.ts:2139-2141

These two comments describe restarts as "max - min of the running total" and totalRestarts as "at its highest in the window", but podRestartsQuery now compiles to greatest(argMax(Value, TimeUnix) - argMin(Value, TimeUnix), 0) and argMax(Value, TimeUnix) (packages/query-engine/src/ch/queries/infra.ts:766-770) — last minus first, and the newest sample. Any reader of the shared domain schema, including whoever renders lastTerminatedReason for the MCP pod view, will take the old wording at face value.

Reword the two comments to the shipped semantics: "Restarts inside the window (last − first of the running total, floored at 0)" and "The running total at the window's newest sample".

🤖 Prompt to fix this finding with an AI agent

Findings from an automated review of commit 46015d9e1b44913b9a85afc6c38b3ed31d9307d3. Verify each one against the current code before changing anything, fix only those that still apply, and keep each fix to the lines it names.

---

F2 · Note · maintainability · packages/domain/src/http/query-engine.ts:2139-2141
`PodRestartsResponse` docs still say max − min while the query returns last − first
These two comments describe `restarts` as "max - min of the running total" and `totalRestarts` as "at its highest in the window", but `podRestartsQuery` now compiles to `greatest(argMax(Value, TimeUnix) - argMin(Value, TimeUnix), 0)` and `argMax(Value, TimeUnix)` (packages/query-engine/src/ch/queries/infra.ts:766-770) — last minus first, and the newest sample. Any reader of the shared domain schema, including whoever renders `lastTerminatedReason` for the MCP pod view, will take the old wording at face value.
Suggested fix: Reword the two comments to the shipped semantics: "Restarts inside the window (last − first of the running total, floored at 0)" and "The running total at the window's newest sample".

Fixed since the last review

• ✅ F1 · restarts overcounts when the kubelet resets the counter inside the window

Production impact

Open errors in the changed files

Issue	Service	Occurrences	File
@maple/web/errors/WarehouseInvalidInputError	maple-web	12	apps/web/src/lib/services/atoms/warehouse-query-atoms.ts

After this merges, Maple checks whether they stop.

Production traffic of the changed files (last 7 days)

File	Calls/day	Busiest operations
apps/api/src/routes/internal/query-engine.http.ts	5	podInfraTimeseries 5/day, 0.0% err
packages/domain/src/http/query-engine.ts	5	podInfraTimeseries 5/day, 0.0% err

What was checked

• greatest(argMax(Value, TimeUnix) - argMin(Value, TimeUnix), 0) is last − first, so the F1 overcount on a counter reset is gone (infra.ts:766)
• Empty podNames drops the name filter instead of emitting IN () (infra.ts:783); the list only mounts the query with a non-empty page (pods/index.tsx:410)
• The new endpoint inherits the group's auto server span (HttpMiddleware.tracer, apps/api/src/http/api-observability.ts:31) and the atom family opens the query span

Observability coverage: 2 of 2 changes observable

Change	Kind	Observable	Evidence
POST /pod-restarts (apps/api internal query-engine API)	server	yes	HttpApiBuilder group middleware HttpMiddleware.tracer stamps the server span (apps/api/src/http/api-observability.ts:31)
podRestarts ClickHouse query via the web atom client	client	yes	makeQueryAtomFamily runs the query on the traced atom runtime and it executes through compiledQuery with context "podRestarts" (warehouse-query-atoms.ts:242-260)

_{46015d9 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}

[maple-review-bot]

Maple review

🟢 Confidence 8/10 · likely safe to merge
Query SQL is pinned by infra.test.ts and the committed catalog baseline; the folding helpers by unit tests, but the new UI wiring has no test.
_{quality 100/100 · no findings · tests partial · risk medium · 2/2 new units observable}

Adds pod restart counts and the last termination reason: a podRestartsQuery, a /pod-restarts endpoint, an atom, and markers on the pod list and pod page. The query is OrgId- and window-scoped, the SQL matches the committed baseline, and F2's max−min doc is corrected. Safe to merge. The open WarehouseInvalidInputError in warehouse-query-atoms.ts is untouched by this change.

• podRestartsQuery counts window restarts as last−first and returns lastTerminatedReason via argMax
• New POST /pod-restarts endpoint, registry query and podRestartsResultAtom
• Pod list shows a restart marker per pod; pod page adds a Restarts tile and container table
• Dead infra presets and OverviewGrain removed to clear knip

Fixed since the last review

• ✅ F2 · PodRestartsResponse docs still say max − min while the query returns last − first

Production impact

Open errors in the changed files

Issue	Service	Occurrences	File
@maple/web/errors/WarehouseInvalidInputError	maple-web	12	apps/web/src/lib/services/atoms/warehouse-query-atoms.ts

After this merges, Maple checks whether they stop.

Production traffic of the changed files (last 7 days)

File	Calls/day	Busiest operations
apps/api/src/routes/internal/query-engine.http.ts	5	podInfraTimeseries 5/day, 0.0% err
packages/domain/src/http/query-engine.ts	5	podInfraTimeseries 5/day, 0.0% err

What was checked

• restarts floors at 0 with greatest(...), so a kubelet counter reset cannot overcount; SQL matches catalog.sql:1467
• Empty podNames skips the filter instead of emitting IN () (infra.ts:783)
• argMax/argMin over a String resource attribute is an existing pattern (catalog.sql:581)

Observability coverage: 2 of 2 changes observable

Change	Kind	Observable	Evidence
POST /pod-restarts (apps/api)	server	yes	Handler added to the existing queryEngine HttpApiBuilder group whose sibling podInfraTimeseries already appears in Maple traces
podRestartsQuery against ClickHouse	database	yes	Runs through runQuery(Queries.podRestarts, ...), the same warehouse path as the other infra queries

_{4c6375b · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}

[maple-review-bot]

Maple review

🟢 Confidence 8/10 · likely safe to merge
The pod-restart query is pinned by the recorded SQL baseline and the folding/tone helpers by unit tests; only the collector's resource attribute itself is unverified here.
_{quality 100/100 · no findings · tests covered · risk medium · 2/2 new units observable}

Adds podRestartsQuery, a POST /pod-restarts endpoint and pod-page UI for restarts and last termination reason. Containment is good and the SQL is pinned by the catalog baseline; safe to merge. The open WarehouseInvalidInputError in warehouse-query-atoms.ts is unrelated and untouched by this diff.

• podRestartsQuery counts window restarts as last minus first, floored at 0, and adds lastTerminatedReason
• POST /pod-restarts plus a podRestarts registry query and API handler
• Pod list shows a per-pod restart marker from a second query over the visible pod names
• Pod page gains a Restarts tile and a per-container restarts table

Production impact

Open errors in the changed files

Issue	Service	Occurrences	File
@maple/web/errors/WarehouseInvalidInputError	maple-web	12	apps/web/src/lib/services/atoms/warehouse-query-atoms.ts

After this merges, Maple checks whether they stop.

Production traffic of the changed files (last 7 days)

File	Calls/day	Busiest operations
apps/api/src/routes/internal/query-engine.http.ts	5	podInfraTimeseries 5/day, 0.0% err
packages/domain/src/http/query-engine.ts	5	podInfraTimeseries 5/day, 0.0% err

What was checked

• Counter resets: greatest(argMax - argMin, 0) cannot overcount, matching the doc at infra.ts:739
• Empty podNames skips the filter instead of emitting IN () (infra.ts:783, infra.test.ts:372)
• ResourceAttributes[...] and argMax per container are pinned verbatim in __sql_baseline__/catalog.sql:1467

Observability coverage: 2 of 2 changes observable

Change	Kind	Observable	Evidence
POST /pod-restarts (query-engine HTTP API)	inbound endpoint	yes	The API registers HttpMiddleware.tracer for all HttpApi routes (apps/api/src/http/api-observability.ts:28-34); siblings like podInfraTimeseries already report ~5 calls/day in Maple.
podRestarts warehouse query	outbound query	yes	Runs through the shared runQuery/runWarehouseQuery path, the same one that reports podInfraTimeseries operations for maple-web.

_{cd60ce5 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}