fix(toolkit-lib): cdk watch loses log events when one log group is throttled - #1992
Merged
Merged
Conversation
…rottled The watch log monitor reads all log groups in parallel with Promise.all. When one FilterLogEvents call failed (usually a ThrottlingException on stacks with many functions), Promise.all rejected and the whole tick was discarded. The other log groups had already moved their start time past the events they read, so those events were never printed. The user only saw "Error occurred while monitoring logs: ThrottlingException: Rate exceeded". Errors are now handled per log group. The events of the other log groups are printed. A throttled log group keeps its start time, so its events are read on the next tick, and the throttle is reported at debug level instead of as an error. Other errors are still reported. Fixes aws#879 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1992 +/- ##
==========================================
+ Coverage 91.47% 91.51% +0.03%
==========================================
Files 80 80
Lines 12675 12675
Branches 1792 1794 +2
==========================================
+ Hits 11595 11599 +4
+ Misses 1042 1038 -4
Partials 38 38
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
mrgrain
approved these changes
Sep 23, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason for this change
cdk watchon stacks with many functions often prints this error (#879, open since 2022, repro repo in the issue):The error message is only part of the problem.
CloudWatchLogEventMonitor.readNewEvents()reads every log group in parallel withPromise.all.readEventsFromLogGroup()moves each group's start time past the events it read, and does that when the call succeeds. When any oneFilterLogEventscall fails,Promise.allrejects andtick()discards the whole result. The groups that succeeded have already moved their start time, so their events are never printed. On a large stack, one throttled group on a tick hides the logs of all the other functions for that tick.Description of changes
readNewEvents()now gives a result per log group ({ events }or{ error }) and does not reject.tick()prints the events of every group that succeeded and reports each error.reportError()helper. AThrottlingExceptionis written at debug level, because the throttled group keeps its start time and its events are read on the next tick, so nothing is lost. Other errors are still reported withCDK_TOOLKIT_E5035, as before.flattenimport was removed.The polling interval and the parallelism are not changed. This PR only fixes the lost events and the error noise for throttles that fix themselves.
Description of how you validated changes
Added two tests in
test/api/logs-monitor/logs-monitor.test.tswith two log groups:group-bis throttled once.group-a's event is printed,group-b's event is printed on the next tick, and no error is shown. Onmain,group-a's event is lost for good, and the output is onlyError occurred while monitoring logs: ThrottlingException: Rate exceeded.group-bfails with another error. The error is still reported, andgroup-a's event is still printed. Onmain, only the error is shown.Both fail on
mainand pass with the fix.logs-monitor,find-cloudwatch-logs,watchanddeploy-trace-logstests pass. Lint and typecheck pass.Checklist
Fixes #879
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license
🤖 Generated with Claude Code