CollectiveX: add NCCL EP as a fourth pluggable EP backend

[Oseltamivir]

Summary

Adds NVIDIA's NCCL EP (nccl/contrib/nccl_ep, consumed as the published nccl4py[cu13]==0.3.1 wheel, libnccl_ep v0.1.0) as CollectiveX's fourth pluggable EP backend, alongside deepep-v2, mori and uccl-ep. Same case/shard codec, same launchers, same seam-contract tests — no new execution path.

• bench/ep_nccl.py — the adapter. NVIDIA-only, cu130, BF16-only (upstream RELEASE.md lists its FP8 machinery unsupported/untested). HT = HIGH_THROUGHPUT/FLAT under the harness's unweighted rank-sum combine; LL = LOW_LATENCY/EXPERT_MAJOR. It forms its own NCCL communicator and uses the torch process group only to broadcast that communicator's unique id, so no MPI dependency is introduced.
• runtime/prepare_backend.sh — nccl_ep_{spec_slug,cache_root,activate,probe,install,prepare}: a pip install into an isolated $root/site keyed by cpu/arch/image/spec under the shared /cx-cache, guarded by flock + a .ready marker, with a node-local fallback for manual runs. Mirrors the existing uccl_* seam.
• runtime/common.sh — pins the wheel spec, plus repo/commit for a from-source fallback that stays deferred. contrib/nccl_ep is absent from the v2.29.x/v2.30.4 release tags, so any such build must use the post-merge master commit recorded here rather than a tag.
• Launchers — launch_single-slurm.sh and launch_gb-nv.sh accept the backend; the GB launcher's pinned-DeepEP source stage becomes conditional, since nccl-ep needs no source tree.
• configs/platform_config.json — h100/h200/b200-dgxc and b300 get nccl-ep:[8]; gb200/gb300 get nccl-ep:[8,16].
• sweep_matrix.py emits BF16 cases only; the sweep workflow gains nccl-ep in its backend dropdown.
• tests/ — rollout-shape and case-argv round-trip coverage extending the existing pattern. Full suite green locally.

Why x86 has no EP16 row

Cross-node EP16 fails inside NCCL EP v0.1.0's GDAKI GIN transport with an illegal memory access at nccl_ep.cc:2884 — identically over RoCE (h100, b200, b300) and IB (h200), through the canonical launcher and rendezvous. It is not adapter-, rendezvous- or selector-fixable, so it is recorded as an unsupported coverage row and never dispatched. EP16 is scheduled only where the world stays inside the scale-up domain (MNNVL/NVL72), which is why gb200/gb300 carry [8, 16].

Validation (on metal, production launchers, not CI)

SKU	EP8 normal	EP16 normal
h100-dgxc	green — roundtrip p50/p99 267.9/298.1 us	n/a (walled)
b300	green — 208.4/230.4 us	n/a (walled)
b200-dgxc	green — 287.8/311.3 us	n/a (walled)
gb200	green — 280.0/673.6 us	green — 352.8/778.1 us
gb300	green — 275.9/341.2 us	green — 344.3/416.3 us

Every leg: status=success, all points correctness-passing across the full decode ladder (T=1..512) and prefill ladder (T=1024..8192). Figures are top-decode-rung roundtrip percentiles. h200-dgxc EP8 was validated in an earlier on-metal pass; its cluster was saturated during the latest one, so it is not re-measured here.

Known issue — the low-latency path wedges on x86

The registry still lists ll_backends nccl-ep:[8] on the x86 SKUs, but every low-latency attempt on h100-dgxc, b300 and b200-dgxc wedges: one rank falls behind, its peers time out their dispatch receives against the compile-time ~100 s EP timeout, and the kernel traps (cudaErrorLaunchFailure, 719). The laggard is stable per node across re-runs.

This is the stale-signal / pipeline-desync race in the wheel's LL count/flag protocol — signal values carry no generation, so a value left over from the previous use of the same parity buffer is bit-identical to a fresh one at a repeating workload. Reported upstream as NVIDIA/nccl#2303 and fixed by NVIDIA/nccl#2306. It cannot be fixed from this tree, which installs the published wheel; the deferred from-source arm above is the only local lever.

Reviewer decision requested: drop or gate those ll_backends rows in a follow-up, or leave them pending a fixed wheel. Note also that the UCCL-EP precedent (#2288) refreshed README.md and docs/methodology.md; this PR does not, and I'm happy to add the backend's documentation rows in a follow-up or here.

[claude]

Beyond the inline findings, I also checked the rank-0-only fallible work (nccl_core.get_unique_id()) before the comm-bootstrap broadcast in ep_nccl.py::_bootstrap_comm for a deadlock if rank 0 throws before reaching dist.broadcast — this is not a new risk introduced by this PR: the same rank-0-does-fallible-setup-then-broadcasts shape already exists in ep_harness.py (line 292), so it's a pre-existing pattern rather than something unique to this adapter.

Extended reasoning...

Ruled out as part of this review pass: the candidate deadlock in ep_nccl.py's _bootstrap_comm (rank 0 could throw before the dist.broadcast call, stranding other ranks) mirrors an existing rank-0-only-then-broadcast pattern already present in ep_harness.py, so it is not a new risk this PR introduces. This is noted for the record only; the PR's own reported bugs (stale docs, and the known-wedging ll_backends rows) are what actually warrant human review before merge.

[claude]

🔴 PR description confirms every low-latency nccl-ep attempt on h100-dgxc, h200-dgxc, b200-dgxc, and b300 wedges (a lagging rank times out its dispatch receive against the ~100s EP timeout and the kernel traps with cudaErrorLaunchFailure/719), yet platform_config.json still enables ll_backends: {"nccl-ep": [8]} on all four SKUs. Since sweep_matrix.py's _ll_runnable reads this map directly and test_nccl_ep_rollout_shape locks the rows in as runnable, merging as-is schedules guaranteed-failing low-latency EP8 legs on real GPU clusters every time nccl-ep or all is swept. The author explicitly asks for a reviewer call on whether to drop or gate these four config rows; recommend gating them out (or dropping the nccl-ep low-latency rows plus the matching test assertion) in this PR rather than merging a coverage row that is known-broken.

Extended reasoning...

What the bug is. configs/platform_config.json adds "ll_backends": {..., "nccl-ep": [8]} to h100-dgxc (diff line ~13), h200-dgxc (~line 37), b200-dgxc (~line 57), and adds a brand-new ll_backends block containing nccl-ep: [8] to b300, which previously had no ll_backends entry at all. These four rows tell the sweep generator that the low-latency (decode) NCCL EP path is runnable at EP8 on every x86 SKU NCCL EP is offered on.

How it is dispatched. sweep_matrix.py::_ll_runnable is a one-line lookup: return ep in platform.get('ll_backends', {}).get(backend, []). resolve_matrix calls this to decide whether to fan a cell out into a low-latency case, and any cell where it returns True is recorded with disposition: "runnable" — meaning it is picked up by collectivex-sweep.yml and actually dispatched to the GPU cluster whenever the workflow is run with backend: nccl-ep or backend: all. The new test test_nccl_ep_rollout_shape in tests/test_matrix.py goes further and asserts ll == {(sku, 8) for sku in {h100-dgxc, h200-dgxc, b200-dgxc, b300}}, which locks this broken low-latency scheduling in as expected/passing behavior rather than flagging it as unsupported.

Why nothing prevents the failure. The PR's own description, in the 'Known issue — the low-latency path wedges on x86' section, states plainly: 'every low-latency attempt on h100-dgxc, b300 and b200-dgxc wedges: one rank falls behind, its peers time out their dispatch receives against the compile-time ~100s EP timeout, and the kernel traps (cudaErrorLaunchFailure, 719)'. The root cause is identified as the stale-signal/pipeline-desync race in NCCL EP's LL count/flag protocol, filed upstream as NVIDIA/nccl#2303 and fixed by NVIDIA/nccl#2306 — a fix that is NOT present in the pinned nccl4py[cu13]==0.3.1 wheel this tree installs (COLLX_NCCL4PY_SPEC in runtime/common.sh). Nothing in ep_nccl.py, the launcher backend-selection logic, or the sweep generator gates this LL path off — the config rows are the only thing standing between the sweep and a guaranteed crash, and they currently say 'go'.

Why this is actionable within this tree. Unlike the upstream wheel bug itself (which cannot be fixed here), dropping or gating the four ll_backends: {"nccl-ep": [8]} rows (and the corresponding assertion in test_nccl_ep_rollout_shape) is a purely local, one-PR change. The author is explicit about this being an open decision: 'Reviewer decision requested: drop or gate those ll_backends rows in a follow-up, or leave them pending a fixed wheel.' Leaving the rows in place as submitted answers that question implicitly (and wrongly) in favor of dispatching the broken path. This also breaks with the precedent already established elsewhere in this same config for known-broken cells — e.g. cross-node EP16 on the same SKUs is recorded as an unsupported coverage row (never dispatched) precisely because it faults reproducibly at nccl_ep.cc:2884; the low-latency EP8 legs deserve the same treatment instead of being marked runnable.

Step-by-step proof. (1) An operator triggers collectivex-sweep.yml with backend: nccl-ep (or all) and default modes (blank = all modes). (2) The setup job runs sweep_matrix.py, which iterates SKUs and for h100-dgxc/h200-dgxc/b200-dgxc/b300 finds ll_backends["nccl-ep"] == [8], so _ll_runnable(platform, "nccl-ep", 8) returns True and a low-latency EP8 case is added to requested_cases with disposition: "runnable" and folded into an executable shard. (3) The sweep job matrix picks up that shard, allocates real GPU nodes via Slurm, builds/activates the nccl4py==0.3.1 wheel, and runs bench/run_ep.py --backend nccl-ep with mode=low-latency. (4) Per the PR's own validation notes, one rank in that run falls behind due to the stale-signal race in the wheel's LL protocol; its peers block on their dispatch receive until the ~100s EP timeout fires, and the kernel traps with cudaErrorLaunchFailure (error 719). (5) The shard leg fails, burning a real multi-minute GPU allocation and a red CI leg — reproducibly, every time this cell is swept, because the wheel's fix (NVIDIA/nccl#2306) is not present.

Suggested fix. Drop the four "nccl-ep": [8] entries from ll_backends (leaving backends for normal/HT mode intact, since HT validated green), and drop or adjust test_nccl_ep_rollout_shape's low-latency assertion accordingly — the same pattern the config already uses to scope out the EP16 cross-node cells as unsupported rather than dispatching known-broken work.

[claude]

🟡 This PR adds nccl-ep as a BF16-only backend (sweep_matrix.py:41) with new backend/ll_backends rows in platform_config.json, but doesn't refresh README.md or docs/methodology.md the way the UCCL-EP PR (#2288) did. As a result, README.md:16 / methodology.md:24 now falsely claim FP8 dispatch happens 'on every backend', and README.md's backend scope table (lines 66-70) plus the ll_backends prose (lines 28-30) omit nccl-ep entirely. This is a pre-existing-pattern gap the author already flagged and offered to fix here or in a follow-up.

Extended reasoning...

What's stale/false: Two related documentation problems land with this PR because it changes backend capability without touching README.md or docs/methodology.md:

1. Active factual inaccuracy. README.md:16 states dispatch precision is "a BF16 control plus an FP8 dispatch on every backend, caller-prequantized in normal mode," and docs/methodology.md:24 says similarly "a BF16 control and an FP8 dispatch (bf16, fp8)." This PR adds nccl-ep as a fourth backend that is BF16-only: sweep_matrix.py sets BACKEND_PRECISIONS["nccl-ep"] = ("bf16",) (diff line 41), and the new test_nccl_ep_rollout_shape test asserts every nccl-ep case emits precision == "bf16" with no FP8 case ever generated. So "an FP8 dispatch on every backend" is no longer true — it's false for one of the four backends CollectiveX now ships.

2. Missing-row staleness. README.md's "Backend | Current scope" table (lines 66-70) and its ll_backends summary prose (lines 28-30, "currently DeepEP V2 EP8 on H100/H200/B200, MoRI EP8 on ..., and UCCL-EP EP8 on H100/H200/B200 only") still enumerate exactly DeepEP V2 / MoRI / UCCL-EP. This PR's configs/platform_config.json changes add nccl-ep to backends on h100/h200/b200-dgxc/b300 (EP8) and gb200/gb300 (EP8+16), and to ll_backends on the four x86 SKUs (EP8) — none of which are reflected in the docs. docs/methodology.md lines 75-79 repeat the same enumeration gap.

Why nothing catches this today: These are prose/table facts in Markdown, not code assertions — there is no test that cross-checks README/methodology text against platform_config.json or sweep_matrix.BACKEND_PRECISIONS, so nothing fails CI when the docs drift from the code they describe.

Precedent: The prior UCCL-EP PR (#2288), which added a new backend in exactly the same shape (new adapter + platform_config rows + precision entry), refreshed both README.md and docs/methodology.md in the same PR. This PR's own description explicitly acknowledges the gap: "the UCCL-EP precedent (#2288) refreshed README.md and docs/methodology.md; this PR does not... I'm happy to add the backend's documentation rows in a follow-up or here."

Step-by-step proof of the false claim:

1. Read README.md:16 — verbatim: "dispatch precision is a swept dimension — a BF16 control plus an FP8 dispatch on every backend, caller-prequantized in normal mode."
2. Read the diff's sweep_matrix.py lines 36-43 — BACKEND_PRECISIONS["nccl-ep"] = ("bf16",), with a comment noting NCCL EP's FP8 machinery is unsupported/untested upstream this release.
3. Read sweep_matrix.py's case-generation loop — for a runnable cell, emit_precisions is filtered to supported = [p for p in SWEEP["precisions"] if p in backend_precisions], so for nccl-ep this is just ["bf16"]; no fp8 case is ever emitted for this backend.
4. Read the new test_nccl_ep_rollout_shape test — it asserts {item["case"]["precision"] for ... if backend == "nccl-ep"} == {"bf16"}, confirming zero FP8 cases exist for nccl-ep across the whole matrix.
5. Therefore "an FP8 dispatch on every backend" is contradicted by the codebase itself post-merge: nccl-ep is a real, sweep-registered backend that never dispatches FP8.

Fix: Update README.md:16 and docs/methodology.md:24 to scope the FP8 claim to the backends that support it (deepep-v2/mori/uccl-ep) rather than "every backend," and add nccl-ep rows to the "Backend | Current scope" table and the ll_backends prose in both files, mirroring what #2288 did for uccl-ep. Given the author already offered to do this in-PR or as a follow-up, either is acceptable — this is a documentation-only gap with no effect on sweep execution or correctness, so it shouldn't block merge.

[Oseltamivir]

Folded in b9019422f — launch_mi-amds.sh now puts the from-source uccl-ep backend cache under COLLX_STAGE_DIR instead of COLLX_SQUASH_DIR.

Root-caused from full-sweep run 30155842613, where both mi355x uccl-ep legs died in ~17s at probe.py's prepare_cache with PermissionError: '/var/lib/squash/.collectivex-backend-cache-<uid>'. This SKU's squash dir is node-local and root-owned (drwxr-xr-x root root on the runner host, drwxrwxrwx on the compute nodes), so it fails twice over: the submit-side mkdir is denied to the runner account, and even as root the directory created on the submit host is not the one the compute node bind-mounts. COLLX_STAGE_DIR is runner-owned and on the cluster's shared filesystem (verified visible from a compute node), and the cache lands beside job_<tag> rather than inside it, so build-once/reuse-per-allocation still holds. Only uccl-ep takes this path — mori ships in the image, which is why mori legs were green on mi355x while both uccl-ep legs failed.

Unrelated to this PR's backend but folded in at the author's request. A fresh full-matrix sweep is running on the branch.