Skip to content

fix(supervisor): stop selecting closed shutdown channel - #1760

Open
theolujay wants to merge 2 commits into
cloudflare:masterfrom
theolujay:master
Open

theolujay wants to merge 2 commits into
cloudflare:masterfrom
theolujay:master

Conversation

@theolujay

@theolujay theolujay commented Oct 8, 2026 •

Copy link
Copy Markdown

Problem

After SIGTERM, Supervisor.Run continues selecting on the closed gracefulShutdownC channel. Since receives from a closed channel are always ready, the supervisor loop spins throughout the graceful shutdown period and consumes about one CPU core.

image

Change

Disable the supervisor's graceful-shutdown select case after its first notification.

Outcome

Observed during a local test with a long-lived SSE request:

State Before After
Idle ~0% CPU ~0% CPU
Streaming ~0% CPU ~0% CPU
Draining after SIGTERM ~100% of one core ~0% CPU

The SSE request completed after SIGTERM, and cloudflared exited after draining.

Reproduction

  1. Run a local origin that streams one SSE event per second for 90 seconds:

    package main
    
    import (
        "fmt"
        "net/http"
        "time"
    )
    
    func main() {
        http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Content-Type", "text/event-stream")
            w.Header().Set("Cache-Control", "no-cache")
            w.Header().Set("Connection", "keep-alive")
            w.WriteHeader(http.StatusOK)
    
            flusher, ok := w.(http.Flusher)
            if !ok {
                return
            }
    
            for i := range 90 {
                if _, err := fmt.Fprintf(w, "data: event %d\n\n", i+1); err != nil {
                    return
                }
                flusher.Flush()
    
                select {
                case <-r.Context().Done():
                    return
                case <-time.After(time.Second):
                }
            }
        })
    
        if err := http.ListenAndServe("127.0.0.1:8000", nil); err != nil {
            panic(err)
        }
    }
  2. Start cloudflared with a Quick Tunnel and its metrics/pprof listener:

    ./cloudflared tunnel \
      --url http://localhost:8000 \
      --grace-period 100s \
      --no-autoupdate \
      --metrics 127.0.0.1:20241
  3. Open the generated trycloudflare.com URL in another terminal, keeping the stream open:

    curl -N https://generated-name.trycloudflare.com/
  4. Send SIGTERM to the cloudflared process while the stream is active:

    kill -TERM <cloudflared-pid>
  5. Observe CPU during the drain with top or by sampling the process CPU time. Before the fix, cloudflared used about one core until shutdown completed; after the fix, it stayed near idle.

  6. Inspect goroutine state during the drain:

    curl 'http://127.0.0.1:20241/debug/pprof/goroutine?debug=2'

    Before the fix, the supervisor goroutine was runnable in Supervisor.Run. After the fix, it was blocked in the main select while waiting for shutdown work to finish.


Closes #1753

I acknowledge PR #1755 covers the same fix and I'm comfortable with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 After SIGTERM, cloudflared tunnel run uses 100% of one CPU core for the whole graceful-shutdown period

1 participant