Skip to content

[https://nvbugs/7004364][fix] Extend GPU lifecycle startup budget #21219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chienchunhung merged 1 commit into from
Oct 30, 2026
Merged

[https://nvbugs/7004364][fix] Extend GPU lifecycle startup budget #21219

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions tests/integration/test_lists/waives.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,10 +180,6 @@ unittest/_torch/test_connector_adp.py SKIP (https://nvbugs/7003528)
unittest/bindings/test_transfer_agent_bindings.py::TestMooncakeFunctionalTransfer::test_mooncake_wait_in_progress_on_zero_timeout SKIP (https://nvbugs/6517836)
unittest/bindings/test_transfer_agent_bindings.py::TestMooncakeFunctionalTransfer::test_mooncake_write_transfer_gpu_tensor SKIP (https://nvbugs/6517836)
unittest/bindings/test_transfer_agent_bindings.py::TestMooncakeFunctionalTransfer::test_mooncake_write_transfer_multiple_chunks SKIP (https://nvbugs/6517836)
unittest/disaggregated/test_transfer_lifecycle_gpu.py::test_transfer_lifecycle_gpu[baseline] SKIP (https://nvbugs/7004364)
unittest/disaggregated/test_transfer_lifecycle_gpu.py::test_transfer_lifecycle_gpu[cancel_before_publication] SKIP (https://nvbugs/7004364)
unittest/disaggregated/test_transfer_lifecycle_gpu.py::test_transfer_lifecycle_gpu[cancel_late_aux] SKIP (https://nvbugs/7011346)
unittest/disaggregated/test_transfer_lifecycle_gpu.py::test_transfer_lifecycle_gpu[cancel_late_kv] SKIP (https://nvbugs/7011346)
unittest/disaggregated/test_transfer_lifecycle_gpu.py::test_transfer_lifecycle_gpu[destination_fatal] SKIP (https://nvbugs/6902353)
unittest/disaggregated/test_transfer_lifecycle_gpu.py::test_transfer_lifecycle_gpu[source_fatal] SKIP (https://nvbugs/6902353)
unittest/disaggregated/test_transfer_lifecycle_gpu.py::test_transfer_lifecycle_gpu[timeout_late_aux] SKIP (https://nvbugs/6902353)
Expand Down
2 changes: 1 addition & 1 deletion tests/unittest/disaggregated/lifecycle_acceptance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ or reused identities mean the original process no longer executes; a live or
unobservable rank fails. This process observation is not GPU/RDMA fencing proof.
The [exit-observer tests](test_transfer_lifecycle_process_exit.py) exercise all
nine scenarios on each rank. The [startup tests](test_transfer_lifecycle_startup.py)
check that both workers and the supervisor share one 120-second startup deadline,
check that both workers and the supervisor share one 600-second startup deadline,
reject readiness observed at or after expiry, and leave relative transfer waits
unchanged; startup failures print bounded worker logs and missing readiness
markers before cleanup.
Expand Down
4 changes: 2 additions & 2 deletions tests/unittest/disaggregated/test_transfer_lifecycle_gpu.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
)
_PHASE_TIMEOUT_S = 30.0
_TIMEOUT_S = 4 * _PHASE_TIMEOUT_S
_WORKER_TIMEOUT_S = 120.0
_WORKER_TIMEOUT_S = 600.0


def _wait(
Expand Down Expand Up @@ -961,7 +961,7 @@ def retired() -> bool:
_record(directory, f"{role}.reused_after_done", stable_outcome=session.status.name, sentinel=77)


@pytest.mark.timeout(600, method="signal")
@pytest.mark.timeout(_WORKER_TIMEOUT_S + 4 * _TIMEOUT_S, method="signal")
@pytest.mark.parametrize("case", _CASES)
def test_transfer_lifecycle_gpu(case: str, tmp_path: Path) -> None:
"""Qualify real GPU/NIXL software lifecycle with four GPUs and separate MPI jobs.
Expand Down
Loading