Skip to content

test: Pin the shutdown budget and the delayed close - #920

Merged
vvoytovych merged 1 commit into
mainfrom
fix/709-shutdown-budget-test
Nov 18, 2026
Merged

vvoytovych merged 1 commit into
mainfrom
fix/709-shutdown-budget-test

Conversation

@vvoytovych

Copy link
Copy Markdown
Contributor

Summary

#919 makes the manager keep serving for 10 seconds after SIGTERM, then stop with a graceful shutdown of at most 30 seconds, inside a grace period of 45 seconds. In the review of #919, ecv found that no test guards this wiring: a deleted delayed close, or other values for the delay, the timeout or the grace period, leave every test green. This PR adds two tests in internal/cmd/manager/shutdown_test.go, and changes no other file.

TestShutdownFitsTheGracePeriod
  grace := terminationGracePeriodSeconds of the Deployment that runs "/network-services manager",
           read from config/manager/manager.yaml
  webhookShutdownDelay > 0
  webhookShutdownDelay + managerGracefulShutdownTimeout < grace          10 s + 30 s < 45 s

TestManagerStartsTheDelayedClose                                         reads manager.go
  shutdown := ctrl.SetupSignalHandler()                                  exactly once
  ctx, stopManager := context.WithCancel(...)                            exactly once
  go webhookServer.CloseConnectionsThenStop(shutdown, webhookShutdownDelay, stopManager)
                                                                         exactly once, in a goroutine

Design choices

  • The wiring test reads the syntax tree of manager.go. The wiring is inline in the command's RunE, and a test cannot run that without a cluster. A behaviour test would need the wiring in a function of its own, which is a change to production code. I kept this PR to tests only. This package already reads Go source in a test (manager_test.go), and so does internal/agent (catalog_test.go).
  • The budget test reads the grace period from the manifest, not from a copy of its value, so a change to the manifest alone also fails it.

Evidence

Before: I broke the wiring in eight ways, one at a time. Each one fails a test:

Change Failing test and message
terminationGracePeriodSeconds: 40 TestShutdownFitsTheGracePeriod: "40s" is not less than "40s"
managerGracefulShutdownTimeout = 40 * time.Second TestShutdownFitsTheGracePeriod: "50s" is not less than "45s"
webhookShutdownDelay = 0 TestShutdownFitsTheGracePeriod: "0" is not positive
The wiring before #919: ctx := ctrl.SetupSignalHandler(), no delayed close TestManagerStartsTheDelayedClose: no context.WithCancel
The go ... CloseConnectionsThenStop(...) line deleted, its two names kept in use TestManagerStartsTheDelayedClose: no delayed close
The delayed close called without go TestManagerStartsTheDelayedClose: no delayed close
time.Second in place of webhookShutdownDelay TestManagerStartsTheDelayedClose: the arguments differ
A function that does not call stopManager TestManagerStartsTheDelayedClose: the arguments differ

A deletion of only the go ... CloseConnectionsThenStop(...) line already breaks the build, because shutdown and stopManager are then unused. So the rows above use forms that compile.

After: both tests pass. make test, make lint and make verify-generated pass on this head.

Watch: this PR changes no code that runs in a cluster, so no metric can show it. The one read is the ci / Build and test check on this PR, which runs both tests.

Merge Danger

Door: two-way

The PR adds one test file and changes no production code.

Blast Radius: tests

A later change to the shutdown values or to the wiring in manager.go must keep both tests green, or change them with it.

Related to #709

The manager keeps serving for webhookShutdownDelay after SIGTERM, then stops
with GracefulShutdownTimeout. Both must end inside terminationGracePeriodSeconds
in config/manager/manager.yaml, or the kubelet kills the pod before it stops.
TestShutdownFitsTheGracePeriod reads the grace period from the manifest and
fails when the sum does not fit, or when the delay is zero.

TestManagerStartsTheDelayedClose reads manager.go and fails when the command
no longer starts CloseConnectionsThenStop in its own goroutine, with the
signal, webhookShutdownDelay and the manager's cancel function.
@vvoytovych vvoytovych self-assigned this Nov 18, 2026
@vvoytovych
vvoytovych marked this pull request as ready for review November 18, 2026 17:49
@vvoytovych
vvoytovych requested a review from a team as a code owner November 18, 2026 17:49
@vvoytovych
vvoytovych requested a review from ecv November 18, 2026 17:50

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

This closes the follow-up from #919. TestShutdownFitsTheGracePeriod reads terminationGracePeriodSeconds from the manager Deployment, so changing the delay, the timeout or the manifest past the 45s budget fails. TestManagerStartsTheDelayedClose parses manager.go and requires one go ...CloseConnectionsThenStop(signal, webhookShutdownDelay, cancel), so dropping the line or reordering its arguments fails. A missing grace period, or no Deployment running the manager, fails loudly rather than passing.

@vvoytovych
vvoytovych merged commit 38f7e45 into main Nov 18, 2026
10 checks passed
@vvoytovych
vvoytovych deleted the fix/709-shutdown-budget-test branch November 18, 2026 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants