feat(daemon): log checkpoint admission delayed behind trace ingestion - #2256
Open
svarlamov-git-ai wants to merge 1 commit into
Open
feat(daemon): log checkpoint admission delayed behind trace ingestion#2256svarlamov-git-ai wants to merge 1 commit into
svarlamov-git-ai wants to merge 1 commit into
Conversation
svarlamov-git-ai
marked this pull request as ready for review
August 30, 2026 23:47
svarlamov-git-ai
force-pushed
the
fix/2252-admission-delay-logging
branch
2 times, most recently
from
August 31, 2026 00:06
8a68652 to
307e7cc
Compare
svarlamov-git-ai
force-pushed
the
fix/2252-admission-delay-logging
branch
2 times, most recently
from
August 31, 2026 00:49
ce8f430 to
c96a620
Compare
svarlamov-git-ai
force-pushed
the
fix/2252-admission-delay-logging
branch
from
August 31, 2026 16:06
c96a620 to
eb43b93
Compare
svarlamov-git-ai
force-pushed
the
fix/2252-admission-delay-logging
branch
from
August 31, 2026 16:55
eb43b93 to
ab6357d
Compare
svarlamov-git-ai
force-pushed
the
fix/2252-admission-delay-logging
branch
from
August 31, 2026 17:17
ab6357d to
8a7a1b4
Compare
Checkpoint admission waits on the trace-ingest watermark; when that watermark stalls, checkpoints sit "received into bounded ingress" with no admission line, and the delay has to be inferred from absent logs (as in #2252, where seq 77-84 were received but never admitted). Log the wait explicitly once it exceeds five seconds, and again at that cadence while it persists, including the receipt sequence, the target and processed ingest sequences, and the total wait so far. Refs #2252 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
svarlamov-git-ai
force-pushed
the
fix/2252-admission-delay-logging
branch
from
August 31, 2026 18:17
8a7a1b4 to
4172f15
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Observability follow-up for #2252 (stacked on the fix PR), implementing the issue's third suggested direction:
Checkpoint admission waits on the trace-ingest watermark. In the incident, checkpoints sat "received into bounded ingress" with no admission line, and the stall had to be inferred from absent logs. Admission now logs a warning once the wait exceeds 5 seconds (
CHECKPOINT_ADMISSION_DELAY_LOG_INTERVAL), repeating at that cadence while it persists:The healthy path is unaffected:
wait_for_trace_ingest_seqchecks the watermark atomically before enrolling in the notify, so when admission completes in milliseconds the 5s timer is never observably involved, and the wrapped wait is safe to cancel/retry (the inner loop re-enrolls before every re-check, so no wakeups are lost; shutdown still exits the loop).The interval is overridable in test builds via
GIT_AI_TEST_CHECKPOINT_ADMISSION_DELAY_LOG_INTERVAL_MSso the test below keeps a wide margin on slow CI runners instead of racing daemon startup against the 5s threshold.Tests (TDD)
delayed_checkpoint_admission_is_logged— stalls the ingest worker with the existingGIT_AI_TEST_TRACE_INGEST_WORKER_START_DELAY_MShook, enqueues synthetic mutating trace frames, submits a checkpoint, and asserts the warning appears. Fails without this change.