Repository navigation
fix(daemon): do not report trace ingest enqueue during shutdown as a worker crash - #69
Draft
posthog[bot] wants to merge 2 commits into
Draft
posthog[bot] wants to merge 2 commits into
posthog[bot] wants to merge 2 commits into
Conversation
…worker crash The trace ingest worker exits on the shutdown signal. A trace connection that is still open during a stop or restart can then send one more payload to a closed channel. enqueue_trace_payload logged this at error level as "worker may have crashed", and the tracing layer sent it to error tracking. When the send fails and the daemon is shutting down, log at debug and return an error without a second request_shutdown. A closed channel without a shutdown request still logs at error level and stops the daemon. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 2b9ec6bd-d43e-438e-83cf-fe138906df3a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 2b9ec6bd-d43e-438e-83cf-fe138906df3a
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
enqueue_trace_payloaddid not checkis_shutting_down(). It loggedtrace ingest queue send failed: worker may have crashedat error level, and the tracing layer turned that log into an exception.Changes
reason = ingest_worker_stopped_for_shutdown) and returns an error. It does not callrequest_shutdownagain.request_shutdown.request_shutdown, worker gonerequest_shutdownrequest_shutdownNote
A sequence fault (
missing_ingest_seq,duplicate_ingest_seq,reorder_buffer_overflow) callsrequest_shutdownbefore the worker exits. Later payloads therefore log at debug. The sequence fault still logs its own error, so error tracking still sees the real fault.Testing
enqueue_after_shutdown_is_not_reported_as_worker_crash: starts the worker, requests shutdown, waits for the channel to close, then enqueues a payload. It checks the shutdown error and that the queued counter is back to 0.enqueue_after_unexpected_worker_exit_reports_crash: closes the channel without a shutdown request. It checks the crash error and that the daemon starts to shut down.cargo test --locked --lib -- daemon::tests::enqueue daemon::tests::request_shutdown: 4 passed.rustfmt --check --edition 2024 src/daemon.rs: clean.cargo clippy --locked --all-targets: no new findings. The existing warnings, for example thefetch_updatedeprecation on current stable, are on main too.task testsuite (the sandbox has notaskrunner). I ran only the daemon unit tests above.Agent context
autter-runtime, but the code is inautter-cli(src/daemon.rs).try_send. While the worker drains, sends must still succeed. The check is only on the failure path.Created with PostHog Desktop from this inbox report.
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.