Skip to content

fix: prevent LogStreamer.Flush deadlock when context is cancelled#313

Draft
wesleyjellis wants to merge 1 commit into
mainfrom
fix/log-streamer-shutdown-deadlock
Draft

fix: prevent LogStreamer.Flush deadlock when context is cancelled#313
wesleyjellis wants to merge 1 commit into
mainfrom
fix/log-streamer-shutdown-deadlock

Conversation

@wesleyjellis

Copy link
Copy Markdown

Problem

LogStreamer.Run exits on either the quit channel or ctx.Done(). Flush unconditionally sends on the unbuffered quit channel, so when the context has been cancelled (SIGINT/SIGTERM while a job is running) there is no receiver and Flush blocks forever.

Downstream effect in jobWorker: the worker goroutine hangs inside Flush, wg.Wait() in doRun never returns, the runner never unregisters, and the process hangs until SIGKILL (the signal handler only cancels the context on the first signal).

Fix

  • Run now closes a done channel on exit.
  • Flush selects between sending quit and observing done, then waits on done — replacing the fragile time.Sleep(200ms) handshake with real synchronization (this also removes a latent data race on the ring buffer between Run and Flush's drain).
  • The wait loop in Flush breaks out early when Run has already exited instead of spinning until the 30s timeout.
  • The final drain now consumes all remaining buffered lines, not just one per stream, since Run may have exited early with complete lines still buffered.

Testing

  • Added TestLogStreamerFlushAfterContextCancel, which deadlocks on main and passes with this change.
  • go test -race ./pkg/... passes.

🤖 Generated with Claude Code

Flush unconditionally sent on the unbuffered quit channel, but Run also
exits when its context is cancelled (SIGINT/SIGTERM during a job). In
that case there is no receiver and Flush blocks forever, hanging the
worker so wg.Wait() never returns and the runner never unregisters.

Run now closes a done channel on exit, and Flush selects between
sending quit and observing done. This also replaces the 200ms
sleep-based handshake with real synchronization, breaks the wait loop
early when Run has already exited, and drains all remaining buffered
lines (not just one per stream) before flushing processors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant