Skip to content

fix(daemon): do not report trace ingest enqueue during shutdown as a worker crash - #69

Draft
posthog[bot] wants to merge 2 commits into
mainfrom
posthog-self-driving/fixdaemon-skip-trace-ingest-enqueue-acd82c
Draft

posthog[bot] wants to merge 2 commits into
mainfrom
posthog-self-driving/fixdaemon-skip-trace-ingest-enqueue-acd82c

Conversation

@posthog

@posthog posthog Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  • A normal daemon stop or restart can send a false "crash" to error tracking. The team then triages an issue that is not a crash.
  • The trace ingest worker exits on the shutdown signal. A trace connection that is still open can then send one more payload to the closed channel.
  • enqueue_trace_payload did not check is_shutting_down(). It logged trace ingest queue send failed: worker may have crashed at error level, and the tracing layer turned that log into an exception.
  • Data loss is at most one late trace payload during a stop. This change does not alter that.

Changes

  • When the send fails and a shutdown is already in progress, the daemon logs at debug (reason = ingest_worker_stopped_for_shutdown) and returns an error. It does not call request_shutdown again.
  • When the channel closes and no shutdown was requested, behavior does not change: error log, "worker may have crashed", and request_shutdown.
  • The caller behavior does not change. The trace connection loop still stops on the returned error.
Case Before After
Payload after request_shutdown, worker gone error log + exception, second request_shutdown debug log, no exception
Worker gone, no shutdown requested error log + exception, request_shutdown same

Note

A sequence fault (missing_ingest_seq, duplicate_ingest_seq, reorder_buffer_overflow) calls request_shutdown before the worker exits. Later payloads therefore log at debug. The sequence fault still logs its own error, so error tracking still sees the real fault.

Testing

  • Added enqueue_after_shutdown_is_not_reported_as_worker_crash: starts the worker, requests shutdown, waits for the channel to close, then enqueues a payload. It checks the shutdown error and that the queued counter is back to 0.
  • Added enqueue_after_unexpected_worker_exit_reports_crash: closes the channel without a shutdown request. It checks the crash error and that the daemon starts to shut down.
  • cargo test --locked --lib -- daemon::tests::enqueue daemon::tests::request_shutdown: 4 passed.
  • rustfmt --check --edition 2024 src/daemon.rs: clean.
  • cargo clippy --locked --all-targets: no new findings. The existing warnings, for example the fetch_update deprecation on current stable, are on main too.
  • Skipped: the full task test suite (the sandbox has no task runner). I ran only the daemon unit tests above.

Agent context

  • The report named autter-runtime, but the code is in autter-cli (src/daemon.rs).
  • I did not add a shutdown check before try_send. While the worker drains, sends must still succeed. The check is only on the failure path.

Created with PostHog Desktop from this inbox report.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

posthog Bot added 2 commits October 9, 2026 20:55
…worker crash

The trace ingest worker exits on the shutdown signal. A trace connection that is still open during a stop or restart can then send one more payload to a closed channel. enqueue_trace_payload logged this at error level as "worker may have crashed", and the tracing layer sent it to error tracking.

When the send fails and the daemon is shutting down, log at debug and return an error without a second request_shutdown. A closed channel without a shutdown request still logs at error level and stops the daemon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: 2b9ec6bd-d43e-438e-83cf-fe138906df3a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: 2b9ec6bd-d43e-438e-83cf-fe138906df3a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants