fix: prevent LogStreamer.Flush deadlock when context is cancelled#313
Draft
wesleyjellis wants to merge 1 commit into
Draft
fix: prevent LogStreamer.Flush deadlock when context is cancelled#313wesleyjellis wants to merge 1 commit into
wesleyjellis wants to merge 1 commit into
Conversation
Flush unconditionally sent on the unbuffered quit channel, but Run also exits when its context is cancelled (SIGINT/SIGTERM during a job). In that case there is no receiver and Flush blocks forever, hanging the worker so wg.Wait() never returns and the runner never unregisters. Run now closes a done channel on exit, and Flush selects between sending quit and observing done. This also replaces the 200ms sleep-based handshake with real synchronization, breaks the wait loop early when Run has already exited, and drains all remaining buffered lines (not just one per stream) before flushing processors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
LogStreamer.Runexits on either thequitchannel orctx.Done().Flushunconditionally sends on the unbufferedquitchannel, so when the context has been cancelled (SIGINT/SIGTERM while a job is running) there is no receiver andFlushblocks forever.Downstream effect in
jobWorker: the worker goroutine hangs insideFlush,wg.Wait()indoRunnever returns, the runner never unregisters, and the process hangs until SIGKILL (the signal handler only cancels the context on the first signal).Fix
Runnow closes adonechannel on exit.Flushselects between sendingquitand observingdone, then waits ondone— replacing the fragiletime.Sleep(200ms)handshake with real synchronization (this also removes a latent data race on the ring buffer betweenRunandFlush's drain).Flushbreaks out early whenRunhas already exited instead of spinning until the 30s timeout.Runmay have exited early with complete lines still buffered.Testing
TestLogStreamerFlushAfterContextCancel, which deadlocks onmainand passes with this change.go test -race ./pkg/...passes.🤖 Generated with Claude Code