knative: kservices: Stop restart when a Pod check fails - #1015
Elvand-Lie wants to merge 1 commit into
Conversation
illume
left a comment
There was a problem hiding this comment.
Thanks for working on this.
The commit messages could use some tidying up to match our contribution guidelines. We use Linux kernel style — the contributing guide has the details, and git log shows good examples.
Commits that need attention
knative: abort KService restart when safety checks fail— Description must start with a capital letter — e.g.frontend: HomeButton: Fix the buttonnotfrontend: HomeButton: fix the button.
Commit guidelines
- Use atomic commits focused on a single change.
- Use the title format
<area>: <Description of changes>— description must start with a capital letter. - Keep the title under 72 characters (soft requirement).
- Explain the intention and why the change is needed.
- Make commit titles meaningful and describe what changed.
- Do not add code that a later commit rewrites; squash or reorder commits instead.
- Do not include
Fixes #NNin commit messages.
Good examples:
frontend: HomeButton: Fix so it navigates to homebackend: config: Add enable-dynamic-clusters flag
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The implementation matches the stated safety requirements and includes focused regression coverage.
Review effort: Balanced
Findings: None
What changed in this PR
Extracts KService Pod restart orchestration into a testable helper and aborts sequential restarts when safety checks fail.
Changes:
- Stops processing after baseline, deletion, or recovery failures.
- Suppresses false success notifications.
- Adds deterministic regression tests for success and failure paths.
| File | Description |
|---|---|
useKServiceActions.tsx |
Delegates restart orchestration to the helper. |
restartKServicePods.ts |
Implements guarded sequential Pod restarts. |
restartKServicePods.test.ts |
Covers restart ordering and safety-gate failures. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Restarting a KService deleted Pods in a loop that kept going after a failure. When deleting a Pod timed out, when its replacement never became Running, or when the delete call itself threw, the loop went on to delete the next Pod and left the service worse off than before the restart. Abort the restart on the first failure and report why, so a partially restarted service is not made worse. The Pod deletion and recovery helpers move into their own module so the sequential delete, the recovery wait, and each abort path can be exercised directly instead of only through the action hook. Adds regressions for recovery timeout, deletion timeout, a throwing delete, an unclearable availability baseline, a vanishing Pod, and an already-terminating Pod, alongside the sequential happy path. Signed-off-by: Elvand Lie Nababan <elvandlie@gmail.com>
0cbed47 to
4700404
Compare
4700404 to
df1b7f4
Compare
|
@illume Rebased and revised to address the review feedback and commit-history requirements. The branch is now a single atomic commit and has been re-verified. Ready for another look. |
Summary
Stop the sequential KService Pod restart as soon as a required safety gate fails.
The action no longer intentionally deletes a later Pod after the previous restart step fails, and it no longer reports a successful restart after a timeout or deletion failure.
Problem
The restart action deletes KService Pods one at a time and waits for each replacement before moving to the next Pod. Previously, deletion errors, deletion-confirmation timeouts, and replacement-recovery timeouts only displayed an error and allowed the loop to continue.
Timeout-only failures could also leave the failure count at zero, causing the action to report
Restart completed successfully.A failed pre-deletion Pod list used an artificial running baseline of zero. Pods that disappeared or became terminating between the initial snapshot and their turn were skipped, allowing later Pods to be deleted without confirming recovered capacity.
Fix
Require every Pod restart step to pass these gates before considering the next Pod:
The restart now stops immediately when a gate fails, when the target Pod disappears, or when it is already terminating. Later Pods remain untouched, the error identifies the failed gate, and the success notification is suppressed.
The orchestration was extracted from
useKServiceActions()into a focused helper so timeout polling and sequential ordering can be tested deterministically with fake timers. The public hook API and Redeploy action remain unchanged.Testing
Added regression and preservation coverage for:
The primary recovery-timeout and absent-Pod regressions fail against the previous behavior and pass with this change.
Verification
npm run tscnpm run lintnpm run formatnpm run format -- --checknpm run test— 5 files, 29 testsnpm run buildgit diff --checkAll checks passed.
Screenshots are not applicable because this change affects restart orchestration and notifications without changing the visual layout.