[chienchunhung]

Summary

Restore CI coverage for the six transfer-lifecycle GPU cases waived under NVBUG 6902353. Targeted CI on this PR's current head executed and passed all six cases with the original 120-second worker-startup timeout.

Scope

Remove only the six case-specific entries in waives.txt: baseline, cancel_before_publication, cancel_late_aux, cancel_late_kv, source_fatal, and timeout_late_aux.
Preserve the destination_fatal waiver added upstream by #20457.
Retain the existing four-GPU DGX B200 test-list entry.
The test harness, assertions, 120-second startup timeout, 600-second pytest timeout, and product code are unchanged.

Verification

Targeted PR-head CI — October 26, 2026, commit c3327dd51466ee52487c4e3709dde534e908fb18: passed.

Trigger: /bot run --only-multi-gpu-test --stage-list "DGX_B200-4_GPUs-PyTorch-1,DGX_B200-4_GPUs-PyTorch-2,DGX_B200-4_GPUs-PyTorch-3".

PR_Github #80663 and L0_MergeRequest_PR #66415 completed successfully. All three selected B200 stages executed successfully in x86_64 Multi-GPU #2938, with completed Slurm allocations and no stage retries. Aggregate JUnit results were 146 passed, 0 failed, 4 skipped across the targeted run.

The lifecycle module ran in DGX_B200-4_GPUs-PyTorch-3 on four B200 GPUs, node nsc-svg-slurm-1-gpu-3, Slurm job 2389979. JUnit results and its captured pytest log agree: 6 passed, 1 skipped in 471.04 seconds, with all seven cases collected.

Lifecycle case	Result	Call duration
baseline	Passed	39.423s
cancel_before_publication	Passed	39.364s
cancel_late_aux	Passed	39.547s
cancel_late_kv	Passed	38.411s
timeout_late_aux	Passed	159.838s
source_fatal	Passed	154.008s
destination_fatal	Skipped — retained waiver from #20457	0s

These are actual case executions, not collection-only or maintenance skips. Call durations include the complete scenario, not just worker startup. The earlier targeted pipeline #66081 executed no tests during B200 maintenance; it is excluded from this verification.

Full pre-merge status: #66415 is intentionally marked “Partly Tested.” The normal blossom-ci merge check remains failing, so this targeted pass does not establish a passing full pre-merge run. The required multi-GPU approval label is present.

Local checks at the same head: pre-commit run --files tests/integration/test_lists/waives.txt and git diff --check passed, including waiver sorting/duplication, formatting, and AST test-list validation. The original DCO sign-off is retained. Rebase onto df4b424b0028a8838effbc4313c6411fb36d299d preserved upstream's destination_fatal waiver. An independent CPU check executed CBTS WaivesRule and selected the lifecycle block and exactly the three B200 stages above.

Historical remote-cluster evidence — October 15, 2026, four B200 GPUs: seven cases × ten repetitions produced 70 unique executions without replays: 68 JUnit/guard-validated passes, plus two completed passing guard records whose final JUnit was lost to ENOSPC. Observed assertion failures, skips, test errors, startup-120s failures, MPI-exit-60s failures, and pytest-600s failures were all zero. Slurm job 5023705 produced 21 validated passes and the two guard-only records; continuation 5025564 ran only the 47 never-started identities, all passing, and completed with exit 0:0. Repetition 4 was split across attempts; this is not 70 fully validated passes or ten uninterrupted pytest invocations. All 70 case-process cleanup guards settled without intervention.

Those offline runs used python3 -m pytest -p lifecycle_guard -vv -ra --tb=long -o junit_family=xunit2 with explicit node IDs. Harness commit dc8451c50864edc07c75dedeafcb91dbb7a5d478 is byte-identical to this PR's harness (SHA256 c51885ea28f18e879f8cbc1477bd9c7c26317887d340abff1715d39013d49409); the installed runtime was da676ffb8b611a1a0a83c862aa094e736f9fe5aa, version 1.4.0rc1.dev202610140007. This is supporting historical evidence from a different runtime/environment, separate from the current PR-head CI result.

Notes

Reverses the six waivers introduced by #20235. The reviewer's first-executed-case/cold-start concern motivated checking actual CI execution. The new run verifies that the six restored cases pass on this head; it does not isolate a causal fix for the historical startup failure or establish that an intermittent failure cannot recur. Intervening code, dependency, or environment changes may explain the different observations. The destination_fatal waiver remains outside this PR's restored coverage.

• Repository PR checklist reviewed as applicable: this change restores existing test coverage and adds no product behavior, API changes, dependencies, ownership changes, or documentation requirements.

[coderabbitai]

No actionable comments were generated in the recent review.

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 362e7144-d597-4d04-adcb-5ba6e7f4a4a6

📥 Commits

Reviewing files that changed from the base of the PR and between ab6f35d and c3327dd.

📒 Files selected for processing: 1

• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes: 1

• tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The integration waiver list retains the destination_fatal skip entry. It no longer includes skip entries for six other GPU transfer lifecycle cases.

Changes

GPU transfer lifecycle tests

Layer / File(s)	Summary
Remove GPU transfer test waivers tests/integration/test_lists/waives.txt	Removed skip entries for baseline, both cancellation cases, source_fatal, and timeout_late_aux. The destination_fatal entry remains.

Priority: Normal

Estimated code review effort: 1 (Trivial) | ~4 minutes

Change: Other

Suggested reviewers: bowenfu

Merge Risk: Minimal · up to c3327

No confirmed issue prevents merging. The required pre-merge GPU CI should run to validate the restored tests on this PR’s build.

Pre-merge checks: 4

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly identifies the NVBugs reference, classifies the change as a chore, and summarizes the primary change: restoring transfer lifecycle GPU test coverage by removing waivers.
Description check		The description clearly explains the scope, motivation, preserved waiver, targeted CI verification, historical evidence, limitations, and checklist status. It provides sufficient test coverage details…

Full details: Description check

Explanation

The description clearly explains the scope, motivation, preserved waiver, targeted CI verification, historical evidence, limitations, and checklist status. It provides sufficient test coverage details, although it uses custom section names instead of the template's exact headings.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[lowsfer]

Heads-up before this merges: in CI, the worker-startup timeout tracked by NVBUG 6902353 consistently hits whichever test_transfer_lifecycle_gpu case runs first in DGX_B200-4_GPUs-PyTorch-3. Unwaiving these six is likely to move the failure back to baseline rather than clear it.

Evidence from x86-64 Multi-GPU runs:

• Before [https://nvbugs/6902353][chore] waive test_transfer_lifecycle_gpu #20235 (e.g. run gh3#2826, 2026-10-13 11:56 UTC): baseline ran first and failed with Timed out: four native MPI ranks at 124.2 s. All six later cases passed, including destination_fatal (159.9 s).
• After [https://nvbugs/6902353][chore] waive test_transfer_lifecycle_gpu #20235 (merged 2026-10-13 16:52 UTC): the six cases were waived, so destination_fatal became the first and only case. It has failed with the same timeout in 16 of 16 runs since 2026-10-13 20:59 UTC, across 14 PRs, and has not passed once (e.g. [None][perf] Do not block the executor on KV bounce recv-region backp… #19940, [None][fix] Enable VMM CUDA IPC in Kubernetes GPU CI #20042, [None][test] Migrate ordinary disaggregated coverage to KV cache manager V2 #20050, [None][feat] Add the PrimTS native sparse MLA FMHA for DeepSeek-V4 generation and enable it by default on SM100 #20209, [None][perf] One-model speculative decoding: host work between steps, worker-produced target logits, KDA drafts' records #19989, [TRTLLM-16992][perf] Use paged attention for MiniMax-M3 dense layers #20159, [None][perf] primsTS dense bf16, qk-bf16/pv-fp8, fp8 fmha context bringup #20006, [None][perf] Fill the KVCM2 host tier progressively #19650). Before [https://nvbugs/6902353][chore] waive test_transfer_lifecycle_gpu #20235 it passed 6 of 6.

That fits a cold-start cost on a fresh CI node: the first launch exceeds the 120 s startup deadline and later launches are warm. It would also explain why repeated runs on a warm remote cluster don't reproduce it.

#20457 waives destination_fatal under this same bug to unblock pre-merge in the meantime. Suggest holding this unwaive until the first-launch startup is fixed, e.g. a longer deadline for the first launch or a per-module warmup. Otherwise baseline will likely start failing every Multi-GPU run again. This PR's own CI may not catch it: a waive-only change typically doesn't exercise the x86-64 Multi-GPU leg.

[chienchunhung]

Heads-up before this merges: in CI, the worker-startup timeout tracked by NVBUG 6902353 consistently hits whichever test_transfer_lifecycle_gpu case runs first in DGX_B200-4_GPUs-PyTorch-3. Unwaiving these six is likely to move the failure back to baseline rather than clear it.

Evidence from x86-64 Multi-GPU runs:

• Before [https://nvbugs/6902353][chore] waive test_transfer_lifecycle_gpu #20235 (e.g. run gh3#2826, 2026-10-13 11:56 UTC): baseline ran first and failed with Timed out: four native MPI ranks at 124.2 s. All six later cases passed, including destination_fatal (159.9 s).
• After [https://nvbugs/6902353][chore] waive test_transfer_lifecycle_gpu #20235 (merged 2026-10-13 16:52 UTC): the six cases were waived, so destination_fatal became the first and only case. It has failed with the same timeout in 16 of 16 runs since 2026-10-13 20:59 UTC, across 14 PRs, and has not passed once (e.g. [None][perf] Do not block the executor on KV bounce recv-region backp… #19940, [None][fix] Enable VMM CUDA IPC in Kubernetes GPU CI #20042, [None][test] Migrate ordinary disaggregated coverage to KV cache manager V2 #20050, [None][feat] Add the PrimTS native sparse MLA FMHA for DeepSeek-V4 generation #20209, [None][perf] One-model speculative decoding: host work between steps, worker-produced target logits, KDA drafts' records #19989, [TRTLLM-16992][perf] Use paged attention for MiniMax-M3 dense layers #20159, [None][perf] primsTS dense bf16, qk-bf16/pv-fp8, fp8 fmha context bringup #20006, [None][perf] Fill the KVCM2 host tier progressively #19650). Before [https://nvbugs/6902353][chore] waive test_transfer_lifecycle_gpu #20235 it passed 6 of 6.

That fits a cold-start cost on a fresh CI node: the first launch exceeds the 120 s startup deadline and later launches are warm. It would also explain why repeated runs on a warm remote cluster don't reproduce it.

#20457 waives destination_fatal under this same bug to unblock pre-merge in the meantime. Suggest holding this unwaive until the first-launch startup is fixed, e.g. a longer deadline for the first launch or a per-module warmup. Otherwise baseline will likely start failing every Multi-GPU run again. This PR's own CI may not catch it: a waive-only change typically doesn't exercise the x86-64 Multi-GPU leg.

Thanks for the notice @lowsfer . Can you provide the reference PR that this PR should wait for?

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #79960 [ run ] triggered by Bot. Commit: c3327dd Link to invocation

[tensorrt-cicd]

PR_Github #79960 [ run ] completed with state FAILURE. Commit: c3327dd
/LLM/main/L0_MergeRequest_PR pipeline #65814 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #80251 [ run ] triggered by Bot. Commit: c3327dd Link to invocation

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #80258 [ run ] triggered by Bot. Commit: c3327dd Link to invocation

[tensorrt-cicd]

PR_Github #80251 [ run ] completed with state ABORTED. Commit: c3327dd

Link to invocation

[tensorrt-cicd]

PR_Github #80258 [ run ] completed with state SUCCESS. Commit: c3327dd
/LLM/main/L0_MergeRequest_PR pipeline #66052 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --only-multi-gpu-test --stage-list "DGX_B200-4_GPUs-PyTorch-1,DGX_B200-4_GPUs-PyTorch-2,DGX_B200-4_GPUs-PyTorch-3"

[tensorrt-cicd]

PR_Github #80289 [ run ] triggered by Bot. Commit: c3327dd Link to invocation

[tensorrt-cicd]

PR_Github #80289 [ run ] completed with state SUCCESS. Commit: c3327dd
/LLM/main/L0_MergeRequest_PR pipeline #66081 (Partly Tested) completed with status: 'SUCCESS'

CI Report

⚠️ Resource Maintenance — Test Coverage Reduced:

Resource maintenance in LLM/main/L0_Test-x86_64-Multi-GPU #2925: B200 multi-GPU stages are paused to relieve the CI queue backlog (109 running / 459 pending on 2026-10-22). Owner: @sharabeshj-nvda. The CI team decides when to lift it.
Affected stages:
DGX_B200-4_GPUs-PyTorch-1
DGX_B200-4_GPUs-PyTorch-2
DGX_B200-4_GPUs-PyTorch-3

Link to invocation

[chienchunhung]

/bot run --only-multi-gpu-test --stage-list "DGX_B200-4_GPUs-PyTorch-1,DGX_B200-4_GPUs-PyTorch-2,DGX_B200-4_GPUs-PyTorch-3"

[tensorrt-cicd]

PR_Github #80663 [ run ] triggered by Bot. Commit: c3327dd Link to invocation

[tensorrt-cicd]

PR_Github #80663 [ run ] completed with state SUCCESS. Commit: c3327dd
/LLM/main/L0_MergeRequest_PR pipeline #66415 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot skip --comment "Un-waived tests are verified in the targeted CI pipeline. "

[tensorrt-cicd]

PR_Github #80739 [ skip ] triggered by Bot. Commit: c3327dd Link to invocation

[tensorrt-cicd]

PR_Github #80739 [ skip ] completed with state SUCCESS. Commit: c3327dd
Skipping testing for commit c3327dd

Link to invocation