Skip to content

<fix>[core]: reuse GroupedConsumeQueue timer @ZSTAC-86636#4490

Open
ZStack-Robot wants to merge 1 commit into
5.5.28from
sync/zstackio/codex/zstac-86636-grouped-consume-queue-timer-5.5.28
Open

<fix>[core]: reuse GroupedConsumeQueue timer @ZSTAC-86636#4490
ZStack-Robot wants to merge 1 commit into
5.5.28from
sync/zstackio/codex/zstac-86636-grouped-consume-queue-timer-5.5.28

Conversation

@ZStack-Robot

Copy link
Copy Markdown
Collaborator

背景 / 根因

GroupedConsumeQueue.start() 原实现每次执行都会 new Timer(),并在 TimerTask finally 中递归调用 start()。每个 Timer 都会创建独立 TimerThread,旧 Timer 只能依赖 GC/Finalizer 清理。

HA 与 LongJob 通知队列在 MN 启动后即开始空转调度,导致 Timer-* 按约 2 个/秒持续创建;当 GC/Finalizer 清理速度跟不上时,会积压 native threads,最终触发 unable to create new native thread

Jira: http://jira.zstack.io/browse/ZSTAC-86636

本次变更

  • 每个 GroupedConsumeQueue 实例复用一个 daemon Timer
  • 使用 timer.schedule(timerTask, 1000, 1000) 周期执行 collectItems(),替代递归 start()
  • 使用 AtomicBoolean 防止重复 start() 创建重复调度。
  • 捕获单次 collect/consume 异常并记录日志,避免 TimerTask 异常退出后队列不再消费。

影响范围

  • 运行时代码:仅修改 core/src/main/java/org/zstack/core/thread/GroupedConsumeQueue.java
  • 业务影响:HA 进度通知、LongJob 进度通知仍按原 timeliness 配置聚合消费;通知内容、分组 key、消费逻辑不变。
  • 线程影响:从每轮创建新的 TimerThread 收敛为每个队列最多一个 daemon TimerThread,消除 Timer-* 持续增长风险。
  • 配置 / DB / API:无变更。

验证

  • git diff --check
  • env JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home PATH=/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/bin:$PATH mvn -pl core -am -DskipTests compile

sync from gitlab !10446

根因:GroupedConsumeQueue.start() 每次执行都会 new Timer 并在 TimerTask finally 中递归调用 start()。每个 Timer 会创建独立 TimerThread,旧 Timer 只能依赖 GC/Finalizer 清理;HA 与 LongJob 通知队列启动后即空转调度,导致 Timer-* 线程持续创建,最终可能触发 unable to create new native thread。

修复方案:每个 GroupedConsumeQueue 实例复用一个 daemon Timer,通过 schedule(timerTask, 1000, 1000) 周期执行 collectItems();使用 AtomicBoolean 防止重复 start;捕获单次 collect/consume 异常并记录日志,避免 TimerTask 异常退出。

Jira: http://jira.zstack.io/browse/ZSTAC-86636

影响范围:仅影响 GroupedConsumeQueue 的后台分组延迟消费调度方式。HA 进度通知和 LongJob 进度通知仍按原有 timeliness 配置聚合消费;线程模型由每轮创建 TimerThread 收敛为每个队列一个 daemon TimerThread,不改变通知内容和分组逻辑。

Verification:
- git diff --check
- env JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home PATH=/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/bin:$PATH mvn -pl core -am -DskipTests compile

Co-Authored-By-AI: true
@coderabbitai

coderabbitai Bot commented Jul 8, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 52 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: ba9a707c-13fe-44e5-b09e-fe5c210f74bf

📥 Commits

Reviewing files that changed from the base of the PR and between 1fde321 and 14e1386.

📒 Files selected for processing (1)
  • core/src/main/java/org/zstack/core/thread/GroupedConsumeQueue.java

Warning

.coderabbit.yaml has a parsing error

The CodeRabbit configuration file in this repository has a parsing error and default settings were used instead. Please fix the error(s) in the configuration file. You can initialize chat with CodeRabbit to get help with the configuration file.

💥 Parsing errors (1)
Could not fetch remote config from http://open.zstack.ai:20001/code-reviews/zstack-cloud.yaml: TimeoutError: The operation was aborted due to timeout
⚙️ Configuration instructions
  • Please see the configuration documentation for more information.
  • You can also validate your configuration using the online YAML validator.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch sync/zstackio/codex/zstac-86636-grouped-consume-queue-timer-5.5.28

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant