[c10d][xccl2] Enroll xccl2 in the shared c10d backend test suites - #12
Open
frost-intel wants to merge 1 commit into
Open
frost-intel wants to merge 1 commit into
frost-intel wants to merge 1 commit into
Conversation
Adds a single BackendConfig entry so xccl2 is exercised by the four suites driven off C10D_BACKENDS (collectives, p2p, process_group, nan_check), plus an xccl2-specific file for the behaviours that have no analogue in the shared harness. Every capability flag on the new entry was measured on hardware rather than assumed: supports_coalescing=True and supports_dropped_p2p_work=True both pass at 2, 4 and 8 ranks. supports_bitwise_reductions stays False because getXcclReduceOpInternal throws for BAND/BOR/BXOR, and float8_dtypes stays empty because getXcclDataTypeInternal has no float8 mapping. supports_collectives_timing and supports_work_sequence_number stay False; per-work timing and sequence numbers are not implemented yet. supports_gather_single stays False even though ProcessGroupXCCL implements gather_single. torch-xpu-ops' TORCH_LIBRARY_IMPL(c10d, XPU) does not register c10d::gather_into_tensor_, so the dispatcher raises NotImplementedError before the backend is reached. The negative branch of test_gather_single now accepts that message as well as Backend's own "does not support gather_single", since a backend can decline either way. detects_nan_in_collectives is a new flag, defaulting True so no existing backend changes behaviour. It is False for xccl2: torch-xpu-ops' check_for_nan kernel signals via a device-side assert(0) that is a no-op on XPU, so a poisoned all_reduce returns cleanly and even a subsequent torch.xpu.synchronize() reports nothing. The three non-detecting nan_check tests still run and pass. Both torch-xpu-ops gaps are outside this stack; flip the flags when they are filled. The harness itself grows a device_module property built on torch.get_device_module, replacing hardcoded torch.cuda calls in _init_pg, test_sync_barrier_blocks_host_on_stream, test_different_group_initialization_order and test_off_by_default_and_removable, and the device property now omits an index for CPU backends. instantiate_backend_tests gained a matching device-count guard for any non-CPU backend. gloo and the nccl backends are unaffected: gloo still reports 29 passed / 8 skipped. The two split_group tests keep their CUDA-only skips; xccl2 has no split() until onecclCommSplit is wired up later in the stack. test_c10d_xccl2.py mirrors test_c10d_nccl2.py, except that test_options_not_shared_with_legacy_backend inverts nccl2's test_shared_options_type: nccl2 reuses core's ProcessGroupNCCL::Options, while the legacy XCCL backend lives in torch-xpu-ops with an unrelated Options, so xccl2 binds its own type and the test guards against the two being conflated. test_backend_version_reported_after_bootstrap covers the new ProcessGroupXCCL2.backend_version property. It runs an all_reduce first, because xccl2 initializes lazily and the version is only known once initXcclResources has run; before that the property is empty by design. Test plan: on a 12-XPU node, test_c10d_xccl2.py is 5/5; the Xccl2 and Gloo classes across the four shared suites are 67 passed, 16 skipped, 0 failed. xccl2 now declares supports_uneven_all_gather, since ProcessGroupXCCL:: all_gather covers the uneven case.
This was referenced Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (bottom → top)
xccl2/01-foundationxccl2/02-apixccl2/03-workxccl2/04-engine-declxccl2/05-engine-implxccl2/06-backend-surfacexccl2/07-build-wiringxccl2/08-testsxccl2/09-integration-testsEach PR is exactly one commit and targets the one below it. Review bottom-up.
Adds a single BackendConfig entry so xccl2 is exercised by the four
suites driven off C10D_BACKENDS (collectives, p2p, process_group,
nan_check), plus an xccl2-specific file for the behaviours that have no
analogue in the shared harness.
Every capability flag on the new entry was measured on hardware rather
than assumed:
supports_coalescing=True and supports_dropped_p2p_work=True both pass
at 2, 4 and 8 ranks.
supports_bitwise_reductions stays False because
getXcclReduceOpInternal throws for BAND/BOR/BXOR, and float8_dtypes
stays empty because getXcclDataTypeInternal has no float8 mapping.
supports_collectives_timing and supports_work_sequence_number stay
False; per-work timing and sequence numbers are not implemented yet.
supports_gather_single stays False even though ProcessGroupXCCL
implements gather_single. torch-xpu-ops' TORCH_LIBRARY_IMPL(c10d, XPU)
does not register c10d::gather_into_tensor_, so the dispatcher raises
NotImplementedError before the backend is reached. The negative branch
of test_gather_single now accepts that message as well as Backend's
own "does not support gather_single", since a backend can decline
either way.
detects_nan_in_collectives is a new flag, defaulting True so no
existing backend changes behaviour. It is False for xccl2:
torch-xpu-ops' check_for_nan kernel signals via a device-side
assert(0) that is a no-op on XPU, so a poisoned all_reduce returns
cleanly and even a subsequent torch.xpu.synchronize() reports nothing.
The three non-detecting nan_check tests still run and pass.
Both torch-xpu-ops gaps are outside this stack; flip the flags when they
are filled.
The harness itself grows a device_module property built on
torch.get_device_module, replacing hardcoded torch.cuda calls in
_init_pg, test_sync_barrier_blocks_host_on_stream,
test_different_group_initialization_order and
test_off_by_default_and_removable, and the device property now omits an
index for CPU backends. instantiate_backend_tests gained a matching
device-count guard for any non-CPU backend. gloo and the nccl backends
are unaffected: gloo still reports 29 passed / 8 skipped.
The two split_group tests keep their CUDA-only skips; xccl2 has no
split() until onecclCommSplit is wired up later in the stack.
test_c10d_xccl2.py mirrors test_c10d_nccl2.py, except that
test_options_not_shared_with_legacy_backend inverts nccl2's
test_shared_options_type: nccl2 reuses core's ProcessGroupNCCL::Options,
while the legacy XCCL backend lives in torch-xpu-ops with an unrelated
Options, so xccl2 binds its own type and the test guards against the two
being conflated.
test_backend_version_reported_after_bootstrap covers the new
ProcessGroupXCCL2.backend_version property. It runs an all_reduce first,
because xccl2 initializes lazily and the version is only known once
initXcclResources has run; before that the property is empty by design.
Test plan: on a 12-XPU node, test_c10d_xccl2.py is 5/5; the Xccl2 and
Gloo classes across the four shared suites are 67 passed, 16 skipped, 0
failed.
xccl2 now declares supports_uneven_all_gather, since ProcessGroupXCCL::
all_gather covers the uneven case.