Skip to content

fix(toolkit-lib): cdk watch loses log events when one log group is throttled - #1992

Merged
aws-cdk-automation merged 1 commit into
aws:mainfrom
Adityaj0:fix/watch-logs-throttling
Sep 23, 2026
Merged

aws-cdk-automation merged 1 commit into
aws:mainfrom
Adityaj0:fix/watch-logs-throttling

Conversation

@Adityaj0

Copy link
Copy Markdown
Contributor

Reason for this change

cdk watch on stacks with many functions often prints this error (#879, open since 2022, repro repo in the issue):

Error occurred while monitoring logs: ThrottlingException: Rate exceeded

The error message is only part of the problem. CloudWatchLogEventMonitor.readNewEvents() reads every log group in parallel with Promise.all. readEventsFromLogGroup() moves each group's start time past the events it read, and does that when the call succeeds. When any one FilterLogEvents call fails, Promise.all rejects and tick() discards the whole result. The groups that succeeded have already moved their start time, so their events are never printed. On a large stack, one throttled group on a tick hides the logs of all the other functions for that tick.

Description of changes

  • readNewEvents() now gives a result per log group ({ events } or { error }) and does not reject. tick() prints the events of every group that succeeded and reports each error.
  • A new reportError() helper. A ThrottlingException is written at debug level, because the throttled group keeps its start time and its events are read on the next tick, so nothing is lost. Other errors are still reported with CDK_TOOLKIT_E5035, as before.
  • The now-unused flatten import was removed.

The polling interval and the parallelism are not changed. This PR only fixes the lost events and the error noise for throttles that fix themselves.

Description of how you validated changes

Added two tests in test/api/logs-monitor/logs-monitor.test.ts with two log groups:

  • group-b is throttled once. group-a's event is printed, group-b's event is printed on the next tick, and no error is shown. On main, group-a's event is lost for good, and the output is only Error occurred while monitoring logs: ThrottlingException: Rate exceeded.
  • group-b fails with another error. The error is still reported, and group-a's event is still printed. On main, only the error is shown.

Both fail on main and pass with the fix. logs-monitor, find-cloudwatch-logs, watch and deploy-trace-logs tests pass. Lint and typecheck pass.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated — not applicable, SDK calls are mocked
  • No manual edits to generated files

Fixes #879

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

🤖 Generated with Claude Code

…rottled

The watch log monitor reads all log groups in parallel with Promise.all.
When one FilterLogEvents call failed (usually a ThrottlingException on
stacks with many functions), Promise.all rejected and the whole tick was
discarded. The other log groups had already moved their start time past
the events they read, so those events were never printed. The user only
saw "Error occurred while monitoring logs: ThrottlingException: Rate
exceeded".

Errors are now handled per log group. The events of the other log groups
are printed. A throttled log group keeps its start time, so its events are
read on the next tick, and the throttle is reported at debug level instead
of as an error. Other errors are still reported.

Fixes aws#879

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.51%. Comparing base (ca64124) to head (ebe985e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1992      +/-   ##
==========================================
+ Coverage   91.47%   91.51%   +0.03%     
==========================================
  Files          80       80              
  Lines       12675    12675              
  Branches     1792     1794       +2     
==========================================
+ Hits        11595    11599       +4     
+ Misses       1042     1038       -4     
  Partials       38       38              
Flag Coverage Δ
suite.unit 91.51% <ø> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@aws-cdk-automation
aws-cdk-automation added this pull request to the merge queue Sep 23, 2026
Merged via the queue into aws:main with commit e4d65d8 Sep 23, 2026
51 checks passed

This branch was successfully deployed

3 active deployments
run-tests — ebe985e0 Deployed Sep 23, 2026 by Adityaj0 via integ_init-templates (init-typescript-app, 24) #6984
integ-approval — ebe985e0 Deployed Sep 23, 2026 by Adityaj0 via prepare #6984
automation — ebe985e0 Deployed Sep 23, 2026 by Adityaj0 via Triage Pull Requests #2132
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

(watch): occasionally throws "Error occurred while monitoring logs: ThrottlingException: Rate exceeded"

4 participants