Skip to content

fix: Close webhook connections before stopping - #919

Merged
vvoytovych merged 2 commits into
mainfrom
fix/709-path-2-review
Nov 18, 2026
Merged

vvoytovych merged 2 commits into
mainfrom
fix/709-path-2-review

Conversation

@vvoytovych

@vvoytovych vvoytovych commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When a manager pod stops, an admission webhook call can fail with EOF: the webhook server closes the API server's keep-alive connection while a call is on its way.

The manager now keeps serving for 10 seconds after SIGTERM, and every webhook response in that time carries Connection: close, so the API server reconnects through the Service, which no longer sends calls to this pod. The manager also reports ready only while its webhook server accepts connections.

SIGTERM
  0 s    every webhook response carries Connection: close
  10 s   the manager stops: controllers, leader lease, webhook server
         (graceful shutdown, at most 30 s, now a named value)
  45 s   grace period ends (was 10 s): 10 + 30 + 5 s of margin

Why

  • On SIGTERM, controller-runtime stops the manager at once, and the webhook server calls Shutdown with no delay (server.go). Shutdown closes idle keep-alive connections at once (server.go), and Go's client names the race: a POST already on the wire when the server closes the connection fails (transport.go).
  • The API server calls a Service webhook over HTTP/1.1 (client.go), and it does not retry a write after a transport error (request.go).
  • HTTP/1.1 asks a server that intends to close a connection to send Connection: close first (RFC 9112, section 9.6). The Kubernetes API server sets the same header in its own shutdown window (with_retry_after.go).
  • Gatekeeper's webhook keeps serving for a delay after SIGTERM, 10 seconds by default (main.go). controller-runtime has no option for this: ✨ Add support for controller-manager webhook shutdown delay kubernetes-sigs/controller-runtime#2601 proposed one and was closed unmerged.
  • The grace period was 10 seconds (manager.yaml), less than the manager's default graceful shutdown of 30 seconds (internal.go).
  • Readiness was a ping, which passes before the webhook server serves (manager.go). The new check dials the webhook port with TLS (server.go).

Design choices

  • One delay in the process, not a preStop sleep. A preStop sleep waits for the Service to drop the pod, but the webhook server still closes its open connections with no warning when the manager stops, so it needs a second window in Go for those. That is two delays and three numbers to tune. One delay in the process covers both, with two numbers: the delay and the grace period. On kind both forms gave the same result, so I took the smaller one.
  • The test setup lives in each test file. Both tests need a serving certificate and a free port, from envtest's WebhookInstallOptions. Each test sets them up in its own _test.go file, which only go test compiles. A shared helper package would be part of the build, and make notice would then list testify in NOTICE. This repository already copies its envtest setup per package (finalizer_test.go, status_test.go), and so does controller-runtime (webhook_suite_test.go). A build tag would also keep a helper out of the build, but every go test, go vet and lint run would then need the tag.

Evidence

  • Before: with the Connection: close header removed, TestServerClosesEachConnectionDuringTheShutdownDelay fails: "Condition never satisfied: a response carries Connection: close". With the readiness check set back to a ping, TestManagerIsReadyOnlyWhileTheWebhookServerServes fails: "the manager is ready before its webhook server serves". After: both pass.
  • On kind with 2 replicas, a Gateway webhook and an HTTPRoute webhook took about 100 calls per second together while I deleted the pod that holds the API server's connections. On 2026-11-17, in 20 of 20 deletes, no call failed. The pod got SIGTERM at once, and the manager stopped 10 seconds later. A form with a 5 second preStop sleep and no Connection: close ran in the same runs. On 2026-11-17 it had a call fail with EOF in 6 of 20 deletes.
  • make test, make lint and make verify-generated pass on this head. NOTICE is unchanged.

Merge Danger

Door: two-way

A revert restores the old shutdown: the 10 second grace period and the ping readiness check.

Blast Radius: shutdown

Each manager pod now takes about 10 seconds longer to stop, and a new leader waits about 10 seconds longer for the lease. A pod that does not stop in 45 seconds is killed, as before at 10.

Not in this change

  • When the server config file changes, the manager exits at once so that it restarts with the new config. That stop does not get the delay yet. I'll follow up on that separately.
  • A test that pins the shutdown budget (delay plus graceful shutdown under the grace period) and the wiring in manager.go. I'll send it as its own PR.

Related to #709

A deleted manager pod stopped its webhook server at once on SIGTERM. The
server closed the API server's idle keep-alive connections, and a call
the API server sent on one of them at that moment failed with EOF. The
manager also reported ready before its webhook server served.

Key changes:
- on SIGTERM, keep serving for 10 s and answer every webhook call in
  that time with Connection: close, so each client reconnects through
  the Service; then stop the manager
- set the managers' graceful shutdown timeout to 30 s as a named value
- raise terminationGracePeriodSeconds from 10 to 45: the 10 s delay,
  the 30 s graceful shutdown and 5 s of margin
- report ready only while the webhook server accepts TLS connections
- share the test webhook server setup in internal/webhook/webhooktest

Related to #709
@vvoytovych
vvoytovych requested a review from a team as a code owner November 18, 2026 09:48
@vvoytovych vvoytovych self-assigned this Nov 18, 2026

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes.

Verify Generated Files is red because of this PR, and main is green on it. Regenerate NOTICE and the shutdown change is otherwise sound.

  • NOTICE is stale: the new non-test package internal/webhook/webhooktest imports testify and envtest, so go-licenses report ./... now lists github.com/stretchr/testify (plus internal/difflib, internal/spew) and reshuffles the k8s.io/apiextensions-apiserver/pkg entry. Run make notice and commit, or keep the helper out of the build graph (duplicate it in the two test packages, or put it behind a build tag). Run https://github.com/datum-cloud/network-services-operator/actions/runs/44279314595; the last three main runs of the same check passed.
  • Shutdown order is correct: on SIGTERM the 10s delay holds ctx, so the manager, leader lease, controllers and metrics server all keep running unchanged for 10s, then stopManager() cancels once. No double-run; the only cost is the new leader waits ~10s longer for the lease release. 10s + 30s GracefulShutdownTimeout = 40s fits the 45s grace period, so only ~5s margin; a second SIGTERM still exits via controller-runtime's handler.
  • Readiness does not flip at SIGTERM; StartedChecker stays ready until the webhook server stops. That is fine here because the kubelet removes the endpoint when the pod is deleted, and Connection: close covers the connections already open. The PR body's "ready only while the server accepts connections" is accurate; just don't read it as "unready during the delay".
  • Race between the delay goroutine and mgr.Start returning: if the group exits for another reason (config change, startup error) the goroutine only leaks until process exit and does no harm.
  • terminationGracePeriodSeconds: 45 is in config/manager/manager.yaml. Fresh clone of infra: no overlay under apps/network-services-operator or channels/edge overrides it for the manager (the only hit is the unrelated downstream-gateway patch at 10).
  • Tamper results with go test -count=1: dropping the Connection: close header fails TestServerClosesEachConnectionDuringTheShutdownDelay; dropping the time.Sleep(delay) fails the same test; swapping the readiness check for healthz.Ping fails TestManagerIsReadyOnlyWhileTheWebhookServerServes.

Follow-ups:

  • Nothing guards the wiring in manager.go: deleting the go webhookServer.CloseConnectionsThenStop(...) line, the 10s/30s constants or the 45s grace period in the manifest leaves every test green. A table test over the constants (delay + graceful timeout < grace) would pin the budget.

@vvoytovych
vvoytovych marked this pull request as draft November 18, 2026 10:52
The webhooktest package was a non-test package, so `go-licenses report ./...`
counted testify in the build and NOTICE had to list it. Each test now sets up
its serving certificate in its own _test.go file, which only `go test`
compiles, as the finalizer and status tests and controller-runtime's own
webhook tests do.
@vvoytovych
vvoytovych force-pushed the fix/709-path-2-review branch from 766574b to b1e00d9 Compare November 18, 2026 15:08
@vvoytovych

Copy link
Copy Markdown
Contributor Author

Thanks for the careful read. I kept the helper out of the build graph by copying it into the two test packages: in b1e00d9 each test sets up its serving certificate in its own _test.go file, and the webhooktest package is gone. NOTICE is the same as on main, and make verify-generated, make test and make lint pass on the new head. I chose the copy over a build tag. This repository already copies its envtest setup per package (finalizer_test.go, status_test.go), and a tag would need -tags on every test and lint run.

I'll send your follow-up as its own PR, with two tests. One keeps the shutdown delay plus GracefulShutdownTimeout under the manifest's grace period. The other fails when the manager no longer starts the delayed close. Could you take another look?

@vvoytovych
vvoytovych requested a review from ecv November 18, 2026 15:59
@vvoytovych
vvoytovych marked this pull request as ready for review November 18, 2026 15:59

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

b1e00d9 closes the hold: the webhooktest helper is gone and both tests build their serving options from envtest inline, so testify and envtest stay out of the non-test build graph and Verify Generated Files passes without a NOTICE change. The tests assert the same behaviour as before.

Follow-ups:

  • The earlier one stands: nothing pins the 10s delay plus 30s graceful timeout under the 45s grace period.

@vvoytovych
vvoytovych merged commit 7f6caf1 into main Nov 18, 2026
10 of 11 checks passed
@vvoytovych
vvoytovych deleted the fix/709-path-2-review branch November 18, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants