fix(daemon): suppress repeated-identical error storms and expected repo-gone log noise - #2269
Open
svarlamov-git-ai wants to merge 2 commits into
Open
fix(daemon): suppress repeated-identical error storms and expected repo-gone log noise#2269svarlamov-git-ai wants to merge 2 commits into
svarlamov-git-ai wants to merge 2 commits into
Conversation
…po-gone noise Persistently broken daemon-side git environments fail every traced command's side-effect pass with the identical error, producing one error log per user git invocation indefinitely. Separately, repos can vanish between trace ingestion and async processing (temp repos created by tooling), and those expected failures were logged at error level. Add an error-log policy for daemon side-effect errors: the first occurrences of a genuine failure still log at error level, repeated identical failures per repo family downgrade to debug with a rate-limited warn summary (at most one per 30 minutes, including the suppressed count), and expected repo-removed conditions log at debug. A success or a differing error re-arms error-level logging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…worktrees Route the checkpoint failure sites through a checkpoint-specific variant that keeps their structured fields (reason, receipt_seq, trace_id), downgrade the redundant per-failure "checkpoint failed" status line to debug (the policy-driven log carries the error at the right level), and classify git errors that name a deleted path in stderr as expected even when the shared family root survives (deleted linked worktrees). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Two classes of daemon error logs are noise rather than actionable failures:
Repeated-identical-failure storms. On a machine whose daemon-side git environment is persistently broken (e.g. a broken Xcode CLT shim, or the daemon being denied read access to an included gitconfig), every traced command's side-effect pass fails with the identical error. Each failure logs at error level, producing an unbounded stream of identical error logs for as long as the environment stays broken.
Expected async-race conditions logged at error level. Git processing is trace2-driven and fully async, so the repo state at processing time need not match state at command time. Traced commands can run outside any repository, or in a temp repo (created by plugins/tooling) that is deleted before async processing. The resulting failures ("No git repository found for path without exec",
fatal: cannot change to '<path>',fatal: not a git repository,failed to stat) are inherent to the design, not errors.Fix
Add
src/daemon/error_log_policy.rsand route the daemon's side-effect error log sites (command apply/side-effect, async side-effect, checkpoint side-effect/watermark) through a sharedrecord_and_log_side_effect_errorhelper:NotFoundcounts), log at debug as "repo removed before processing" instead of error. The trace-ingest worker's error site applies the same classification for repo-discovery misses.Intentionally left loud: panic sites (genuine bugs) and test-only completion-log write failures still log unconditionally at error level, and the ingest-site downgrade only applies to the no-repo discovery miss (traced commands legitimately run outside repos constantly, so no error budget there).
Testing
Instant, argv/OID-varying storm keys, idle eviction) and the expected-condition classifier (positive and negative cases; genuine git errors stay at error level).task fmt,task lint, fulltask test(lib + integration) pass locally; the two pre-existing env-dependent lib test failures on this machine reproduce identically onmain.🤖 Generated with Claude Code