Skip to content

feat(daemon): log checkpoint admission delayed behind trace ingestion - #2256

Open
svarlamov-git-ai wants to merge 1 commit into
fix/2252-decouple-side-effect-drainsfrom
fix/2252-admission-delay-logging
Open

feat(daemon): log checkpoint admission delayed behind trace ingestion#2256
svarlamov-git-ai wants to merge 1 commit into
fix/2252-decouple-side-effect-drainsfrom
fix/2252-admission-delay-logging

Conversation

@svarlamov-git-ai

@svarlamov-git-ai svarlamov-git-ai commented Aug 30, 2026

Copy link
Copy Markdown

Summary

Observability follow-up for #2252 (stacked on the fix PR), implementing the issue's third suggested direction:

Emit an explicit log line when ingress admission is delayed beyond a threshold (seq 77-84 above had to be inferred from the absence of admission lines).

Checkpoint admission waits on the trace-ingest watermark. In the incident, checkpoints sat "received into bounded ingress" with no admission line, and the stall had to be inferred from absent logs. Admission now logs a warning once the wait exceeds 5 seconds (CHECKPOINT_ADMISSION_DELAY_LOG_INTERVAL), repeating at that cadence while it persists:

WARN checkpoint admission delayed waiting for trace ingestion component="daemon" phase="checkpoint_admission" reason="admission_delayed_by_trace_ingest" receipt_seq=77 trace_ingest_target=1042 processed_trace_ingest_seq=1017 waited_ms=5000

The healthy path is unaffected: wait_for_trace_ingest_seq checks the watermark atomically before enrolling in the notify, so when admission completes in milliseconds the 5s timer is never observably involved, and the wrapped wait is safe to cancel/retry (the inner loop re-enrolls before every re-check, so no wakeups are lost; shutdown still exits the loop).

The interval is overridable in test builds via GIT_AI_TEST_CHECKPOINT_ADMISSION_DELAY_LOG_INTERVAL_MS so the test below keeps a wide margin on slow CI runners instead of racing daemon startup against the 5s threshold.

Tests (TDD)

  • delayed_checkpoint_admission_is_logged — stalls the ingest worker with the existing GIT_AI_TEST_TRACE_INGEST_WORKER_START_DELAY_MS hook, enqueues synthetic mutating trace frames, submits a checkpoint, and asserts the warning appears. Fails without this change.

@svarlamov-git-ai
svarlamov-git-ai marked this pull request as ready for review August 30, 2026 23:47
devin-ai-integration[bot]

This comment was marked as resolved.

@svarlamov-git-ai
svarlamov-git-ai force-pushed the fix/2252-admission-delay-logging branch 2 times, most recently from 8a68652 to 307e7cc Compare August 31, 2026 00:06
devin-ai-integration[bot]

This comment was marked as resolved.

@svarlamov-git-ai
svarlamov-git-ai force-pushed the fix/2252-admission-delay-logging branch 2 times, most recently from ce8f430 to c96a620 Compare August 31, 2026 00:49
@svarlamov-git-ai
svarlamov-git-ai force-pushed the fix/2252-admission-delay-logging branch from c96a620 to eb43b93 Compare August 31, 2026 16:06
@svarlamov-git-ai
svarlamov-git-ai force-pushed the fix/2252-admission-delay-logging branch from eb43b93 to ab6357d Compare August 31, 2026 16:55
@svarlamov-git-ai
svarlamov-git-ai force-pushed the fix/2252-admission-delay-logging branch from ab6357d to 8a7a1b4 Compare August 31, 2026 17:17
Checkpoint admission waits on the trace-ingest watermark; when that
watermark stalls, checkpoints sit "received into bounded ingress" with
no admission line, and the delay has to be inferred from absent logs
(as in #2252, where seq 77-84 were received but never admitted). Log
the wait explicitly once it exceeds five seconds, and again at that
cadence while it persists, including the receipt sequence, the target
and processed ingest sequences, and the total wait so far.

Refs #2252

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@svarlamov-git-ai
svarlamov-git-ai force-pushed the fix/2252-admission-delay-logging branch from 8a7a1b4 to 4172f15 Compare August 31, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants