Skip to content

knative: kservices: Stop restart when a Pod check fails - #1015

Open
Elvand-Lie wants to merge 1 commit into
headlamp-k8s:mainfrom
Elvand-Lie:fix/knative-restart-abort-on-recovery-failure
Open

Elvand-Lie wants to merge 1 commit into
headlamp-k8s:mainfrom
Elvand-Lie:fix/knative-restart-abort-on-recovery-failure

Conversation

@Elvand-Lie

Copy link
Copy Markdown

Summary

Stop the sequential KService Pod restart as soon as a required safety gate fails.

The action no longer intentionally deletes a later Pod after the previous restart step fails, and it no longer reports a successful restart after a timeout or deletion failure.

Problem

The restart action deletes KService Pods one at a time and waits for each replacement before moving to the next Pod. Previously, deletion errors, deletion-confirmation timeouts, and replacement-recovery timeouts only displayed an error and allowed the loop to continue.

Timeout-only failures could also leave the failure count at zero, causing the action to report Restart completed successfully.

A failed pre-deletion Pod list used an artificial running baseline of zero. Pods that disappeared or became terminating between the initial snapshot and their turn were skipped, allowing later Pods to be deleted without confirming recovered capacity.

Fix

Require every Pod restart step to pass these gates before considering the next Pod:

  • establish the pre-deletion running-Pod baseline;
  • complete the deletion request;
  • confirm that the deleted Pod disappeared;
  • confirm recovery to at least the previous running count.

The restart now stops immediately when a gate fails, when the target Pod disappears, or when it is already terminating. Later Pods remain untouched, the error identifies the failed gate, and the success notification is suppressed.

The orchestration was extracted from useKServiceActions() into a focused helper so timeout polling and sequential ordering can be tested deterministically with fake timers. The public hook API and Redeploy action remain unchanged.

Testing

Added regression and preservation coverage for:

  • replacement recovery timing out;
  • deletion confirmation timing out;
  • the deletion request throwing;
  • successful sequential restart of two Pods;
  • failure to establish the running-Pod baseline;
  • the target Pod disappearing before deletion;
  • the target Pod already terminating.

The primary recovery-timeout and absent-Pod regressions fail against the previous behavior and pass with this change.

Verification

  • npm run tsc
  • npm run lint
  • npm run format
  • npm run format -- --check
  • npm run test — 5 files, 29 tests
  • npm run build
  • git diff --check

All checks passed.

Screenshots are not applicable because this change affects restart orchestration and notifications without changing the visual layout.

@illume illume left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this.

The commit messages could use some tidying up to match our contribution guidelines. We use Linux kernel style — the contributing guide has the details, and git log shows good examples.

Commits that need attention
  • knative: abort KService restart when safety checks fail — Description must start with a capital letter — e.g. frontend: HomeButton: Fix the button not frontend: HomeButton: fix the button.
Commit guidelines
  • Use atomic commits focused on a single change.
  • Use the title format <area>: <Description of changes> — description must start with a capital letter.
  • Keep the title under 72 characters (soft requirement).
  • Explain the intention and why the change is needed.
  • Make commit titles meaningful and describe what changed.
  • Do not add code that a later commit rewrites; squash or reorder commits instead.
  • Do not include Fixes #NN in commit messages.

Good examples:

  • frontend: HomeButton: Fix so it navigates to home
  • backend: config: Add enable-dynamic-clusters flag

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation matches the stated safety requirements and includes focused regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Extracts KService Pod restart orchestration into a testable helper and aborts sequential restarts when safety checks fail.

Changes:

  • Stops processing after baseline, deletion, or recovery failures.
  • Suppresses false success notifications.
  • Adds deterministic regression tests for success and failure paths.
File Description
useKServiceActions.tsx Delegates restart orchestration to the helper.
restartKServicePods.ts Implements guarded sequential Pod restarts.
restartKServicePods.test.ts Covers restart ordering and safety-gate failures.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Restarting a KService deleted Pods in a loop that kept going after a
failure. When deleting a Pod timed out, when its replacement never became
Running, or when the delete call itself threw, the loop went on to delete
the next Pod and left the service worse off than before the restart.

Abort the restart on the first failure and report why, so a partially
restarted service is not made worse. The Pod deletion and recovery
helpers move into their own module so the sequential delete, the
recovery wait, and each abort path can be exercised directly instead of
only through the action hook.

Adds regressions for recovery timeout, deletion timeout, a throwing
delete, an unclearable availability baseline, a vanishing Pod, and an
already-terminating Pod, alongside the sequential happy path.

Signed-off-by: Elvand Lie Nababan <elvandlie@gmail.com>
@Elvand-Lie Elvand-Lie changed the title knative: abort KService restart when safety checks fail knative: kservices: Stop restart when a Pod check fails Sep 29, 2026
@Elvand-Lie
Elvand-Lie force-pushed the fix/knative-restart-abort-on-recovery-failure branch from 0cbed47 to 4700404 Compare September 30, 2026 15:28
@Elvand-Lie
Elvand-Lie force-pushed the fix/knative-restart-abort-on-recovery-failure branch from 4700404 to df1b7f4 Compare September 30, 2026 15:49
@Elvand-Lie

Copy link
Copy Markdown
Author

@illume Rebased and revised to address the review feedback and commit-history requirements. The branch is now a single atomic commit and has been re-verified. Ready for another look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants