Repository navigation
fix: Close webhook connections before stopping - #919
Conversation
A deleted manager pod stopped its webhook server at once on SIGTERM. The server closed the API server's idle keep-alive connections, and a call the API server sent on one of them at that moment failed with EOF. The manager also reported ready before its webhook server served. Key changes: - on SIGTERM, keep serving for 10 s and answer every webhook call in that time with Connection: close, so each client reconnects through the Service; then stop the manager - set the managers' graceful shutdown timeout to 30 s as a named value - raise terminationGracePeriodSeconds from 10 to 45: the 10 s delay, the 30 s graceful shutdown and 5 s of margin - report ready only while the webhook server accepts TLS connections - share the test webhook server setup in internal/webhook/webhooktest Related to #709
ecv
left a comment
There was a problem hiding this comment.
Request changes.
Verify Generated Files is red because of this PR, and main is green on it. Regenerate NOTICE and the shutdown change is otherwise sound.
- NOTICE is stale: the new non-test package
internal/webhook/webhooktestimports testify and envtest, sogo-licenses report ./...now listsgithub.com/stretchr/testify(plusinternal/difflib,internal/spew) and reshuffles thek8s.io/apiextensions-apiserver/pkgentry. Runmake noticeand commit, or keep the helper out of the build graph (duplicate it in the two test packages, or put it behind a build tag). Run https://github.com/datum-cloud/network-services-operator/actions/runs/44279314595; the last three main runs of the same check passed. - Shutdown order is correct: on SIGTERM the 10s delay holds
ctx, so the manager, leader lease, controllers and metrics server all keep running unchanged for 10s, thenstopManager()cancels once. No double-run; the only cost is the new leader waits ~10s longer for the lease release. 10s + 30sGracefulShutdownTimeout= 40s fits the 45s grace period, so only ~5s margin; a second SIGTERM still exits via controller-runtime's handler. - Readiness does not flip at SIGTERM;
StartedCheckerstays ready until the webhook server stops. That is fine here because the kubelet removes the endpoint when the pod is deleted, andConnection: closecovers the connections already open. The PR body's "ready only while the server accepts connections" is accurate; just don't read it as "unready during the delay". - Race between the delay goroutine and
mgr.Startreturning: if the group exits for another reason (config change, startup error) the goroutine only leaks until process exit and does no harm. terminationGracePeriodSeconds: 45is inconfig/manager/manager.yaml. Fresh clone of infra: no overlay underapps/network-services-operatororchannels/edgeoverrides it for the manager (the only hit is the unrelated downstream-gateway patch at 10).- Tamper results with
go test -count=1: dropping theConnection: closeheader failsTestServerClosesEachConnectionDuringTheShutdownDelay; dropping thetime.Sleep(delay)fails the same test; swapping the readiness check forhealthz.PingfailsTestManagerIsReadyOnlyWhileTheWebhookServerServes.
Follow-ups:
- Nothing guards the wiring in
manager.go: deleting thego webhookServer.CloseConnectionsThenStop(...)line, the 10s/30s constants or the 45s grace period in the manifest leaves every test green. A table test over the constants (delay + graceful timeout < grace) would pin the budget.
The webhooktest package was a non-test package, so `go-licenses report ./...` counted testify in the build and NOTICE had to list it. Each test now sets up its serving certificate in its own _test.go file, which only `go test` compiles, as the finalizer and status tests and controller-runtime's own webhook tests do.
766574b to
b1e00d9
Compare
|
Thanks for the careful read. I kept the helper out of the build graph by copying it into the two test packages: in b1e00d9 each test sets up its serving certificate in its own I'll send your follow-up as its own PR, with two tests. One keeps the shutdown delay plus |
ecv
left a comment
There was a problem hiding this comment.
Approve.
b1e00d9 closes the hold: the webhooktest helper is gone and both tests build their serving options from envtest inline, so testify and envtest stay out of the non-test build graph and Verify Generated Files passes without a NOTICE change. The tests assert the same behaviour as before.
Follow-ups:
- The earlier one stands: nothing pins the 10s delay plus 30s graceful timeout under the 45s grace period.
Summary
When a manager pod stops, an admission webhook call can fail with EOF: the webhook server closes the API server's keep-alive connection while a call is on its way.
The manager now keeps serving for 10 seconds after SIGTERM, and every webhook response in that time carries
Connection: close, so the API server reconnects through the Service, which no longer sends calls to this pod. The manager also reports ready only while its webhook server accepts connections.Why
Shutdownwith no delay (server.go).Shutdowncloses idle keep-alive connections at once (server.go), and Go's client names the race: a POST already on the wire when the server closes the connection fails (transport.go).Connection: closefirst (RFC 9112, section 9.6). The Kubernetes API server sets the same header in its own shutdown window (with_retry_after.go).Design choices
preStopsleep. ApreStopsleep waits for the Service to drop the pod, but the webhook server still closes its open connections with no warning when the manager stops, so it needs a second window in Go for those. That is two delays and three numbers to tune. One delay in the process covers both, with two numbers: the delay and the grace period. On kind both forms gave the same result, so I took the smaller one.WebhookInstallOptions. Each test sets them up in its own_test.gofile, which onlygo testcompiles. A shared helper package would be part of the build, andmake noticewould then list testify in NOTICE. This repository already copies its envtest setup per package (finalizer_test.go, status_test.go), and so does controller-runtime (webhook_suite_test.go). A build tag would also keep a helper out of the build, but everygo test,go vetand lint run would then need the tag.Evidence
Connection: closeheader removed,TestServerClosesEachConnectionDuringTheShutdownDelayfails: "Condition never satisfied: a response carries Connection: close". With the readiness check set back to a ping,TestManagerIsReadyOnlyWhileTheWebhookServerServesfails: "the manager is ready before its webhook server serves". After: both pass.preStopsleep and noConnection: closeran in the same runs. On 2026-11-17 it had a call fail with EOF in 6 of 20 deletes.make test,make lintandmake verify-generatedpass on this head. NOTICE is unchanged.Merge Danger
Door: two-way
A revert restores the old shutdown: the 10 second grace period and the ping readiness check.
Blast Radius: shutdown
Each manager pod now takes about 10 seconds longer to stop, and a new leader waits about 10 seconds longer for the lease. A pod that does not stop in 45 seconds is killed, as before at 10.
Not in this change
manager.go. I'll send it as its own PR.Related to #709