Skip to content

[c10d][xccl2] Enroll xccl2 in the shared c10d backend test suites - #12

Open
frost-intel wants to merge 1 commit into
xccl2/07-build-wiringfrom
xccl2/08-tests
Open

frost-intel wants to merge 1 commit into
xccl2/07-build-wiringfrom
xccl2/08-tests

Conversation

@frost-intel

@frost-intel frost-intel commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Stack (bottom → top)

PR Branch
#5 xccl2/01-foundation
#6 xccl2/02-api
#7 xccl2/03-work
#8 xccl2/04-engine-decl
#9 xccl2/05-engine-impl
#10 xccl2/06-backend-surface
#11 xccl2/07-build-wiring
#12 xccl2/08-tests
#3 xccl2/09-integration-tests

Each PR is exactly one commit and targets the one below it. Review bottom-up.


Adds a single BackendConfig entry so xccl2 is exercised by the four
suites driven off C10D_BACKENDS (collectives, p2p, process_group,
nan_check), plus an xccl2-specific file for the behaviours that have no
analogue in the shared harness.

Every capability flag on the new entry was measured on hardware rather
than assumed:

supports_coalescing=True and supports_dropped_p2p_work=True both pass
at 2, 4 and 8 ranks.

supports_bitwise_reductions stays False because
getXcclReduceOpInternal throws for BAND/BOR/BXOR, and float8_dtypes
stays empty because getXcclDataTypeInternal has no float8 mapping.

supports_collectives_timing and supports_work_sequence_number stay
False; per-work timing and sequence numbers are not implemented yet.

supports_gather_single stays False even though ProcessGroupXCCL
implements gather_single. torch-xpu-ops' TORCH_LIBRARY_IMPL(c10d, XPU)
does not register c10d::gather_into_tensor_, so the dispatcher raises
NotImplementedError before the backend is reached. The negative branch
of test_gather_single now accepts that message as well as Backend's
own "does not support gather_single", since a backend can decline
either way.

detects_nan_in_collectives is a new flag, defaulting True so no
existing backend changes behaviour. It is False for xccl2:
torch-xpu-ops' check_for_nan kernel signals via a device-side
assert(0) that is a no-op on XPU, so a poisoned all_reduce returns
cleanly and even a subsequent torch.xpu.synchronize() reports nothing.
The three non-detecting nan_check tests still run and pass.

Both torch-xpu-ops gaps are outside this stack; flip the flags when they
are filled.

The harness itself grows a device_module property built on
torch.get_device_module, replacing hardcoded torch.cuda calls in
_init_pg, test_sync_barrier_blocks_host_on_stream,
test_different_group_initialization_order and
test_off_by_default_and_removable, and the device property now omits an
index for CPU backends. instantiate_backend_tests gained a matching
device-count guard for any non-CPU backend. gloo and the nccl backends
are unaffected: gloo still reports 29 passed / 8 skipped.

The two split_group tests keep their CUDA-only skips; xccl2 has no
split() until onecclCommSplit is wired up later in the stack.

test_c10d_xccl2.py mirrors test_c10d_nccl2.py, except that
test_options_not_shared_with_legacy_backend inverts nccl2's
test_shared_options_type: nccl2 reuses core's ProcessGroupNCCL::Options,
while the legacy XCCL backend lives in torch-xpu-ops with an unrelated
Options, so xccl2 binds its own type and the test guards against the two
being conflated.

test_backend_version_reported_after_bootstrap covers the new
ProcessGroupXCCL2.backend_version property. It runs an all_reduce first,
because xccl2 initializes lazily and the version is only known once
initXcclResources has run; before that the property is empty by design.

Test plan: on a 12-XPU node, test_c10d_xccl2.py is 5/5; the Xccl2 and
Gloo classes across the four shared suites are 67 passed, 16 skipped, 0
failed.

xccl2 now declares supports_uneven_all_gather, since ProcessGroupXCCL::
all_gather covers the uneven case.

Adds a single BackendConfig entry so xccl2 is exercised by the four
suites driven off C10D_BACKENDS (collectives, p2p, process_group,
nan_check), plus an xccl2-specific file for the behaviours that have no
analogue in the shared harness.

Every capability flag on the new entry was measured on hardware rather
than assumed:

  supports_coalescing=True and supports_dropped_p2p_work=True both pass
  at 2, 4 and 8 ranks.

  supports_bitwise_reductions stays False because
  getXcclReduceOpInternal throws for BAND/BOR/BXOR, and float8_dtypes
  stays empty because getXcclDataTypeInternal has no float8 mapping.

  supports_collectives_timing and supports_work_sequence_number stay
  False; per-work timing and sequence numbers are not implemented yet.

  supports_gather_single stays False even though ProcessGroupXCCL
  implements gather_single. torch-xpu-ops' TORCH_LIBRARY_IMPL(c10d, XPU)
  does not register c10d::gather_into_tensor_, so the dispatcher raises
  NotImplementedError before the backend is reached. The negative branch
  of test_gather_single now accepts that message as well as Backend's
  own "does not support gather_single", since a backend can decline
  either way.

  detects_nan_in_collectives is a new flag, defaulting True so no
  existing backend changes behaviour. It is False for xccl2:
  torch-xpu-ops' check_for_nan kernel signals via a device-side
  assert(0) that is a no-op on XPU, so a poisoned all_reduce returns
  cleanly and even a subsequent torch.xpu.synchronize() reports nothing.
  The three non-detecting nan_check tests still run and pass.

Both torch-xpu-ops gaps are outside this stack; flip the flags when they
are filled.

The harness itself grows a device_module property built on
torch.get_device_module, replacing hardcoded torch.cuda calls in
_init_pg, test_sync_barrier_blocks_host_on_stream,
test_different_group_initialization_order and
test_off_by_default_and_removable, and the device property now omits an
index for CPU backends. instantiate_backend_tests gained a matching
device-count guard for any non-CPU backend. gloo and the nccl backends
are unaffected: gloo still reports 29 passed / 8 skipped.

The two split_group tests keep their CUDA-only skips; xccl2 has no
split() until onecclCommSplit is wired up later in the stack.

test_c10d_xccl2.py mirrors test_c10d_nccl2.py, except that
test_options_not_shared_with_legacy_backend inverts nccl2's
test_shared_options_type: nccl2 reuses core's ProcessGroupNCCL::Options,
while the legacy XCCL backend lives in torch-xpu-ops with an unrelated
Options, so xccl2 binds its own type and the test guards against the two
being conflated.

test_backend_version_reported_after_bootstrap covers the new
ProcessGroupXCCL2.backend_version property. It runs an all_reduce first,
because xccl2 initializes lazily and the version is only known once
initXcclResources has run; before that the property is empty by design.

Test plan: on a 12-XPU node, test_c10d_xccl2.py is 5/5; the Xccl2 and
Gloo classes across the four shared suites are 67 passed, 16 skipped, 0
failed.

xccl2 now declares supports_uneven_all_gather, since ProcessGroupXCCL::
all_gather covers the uneven case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant