Skip to content

feat: support multinode for JSON output (rebase of #54 onto v0.10) - #72

Closed
jingxu9x wants to merge 1 commit into
NVIDIA:mainfrom
jingxu9x:jingxuz/multinode-json-output
Closed

jingxu9x wants to merge 1 commit into
NVIDIA:mainfrom
jingxu9x:jingxuz/multinode-json-output

Conversation

@jingxu9x

Copy link
Copy Markdown

Rebases #54 onto current main (v0.10) and resolves the conflicts that made it un-mergeable. Original work by @shadychan (credited via Co-authored-by); this supersedes #54.

Why

On multinode --json runs, output->recordDevices(worldSize) reached JsonOutput::recordDevices, which iterated the argument as a local device count and called cuDeviceGet past the local GPU count -> CUDA_ERROR_INVALID_DEVICE. #54 fixed the JSON multinode path but went stale against v0.10, which meanwhile unified the multinode/single-node output paths and added --pair/UUID handling plus a per-node "more processes than GPUs" warning.

What changed vs #54

  • Rebased onto v0.10; kept all of main's newer logic (--pair/UUID selection, the per-node oversubscription warning). Refactored printGPUsMultinode / printGPUs into exchangeDeviceInfo + a single recordDevices.
  • Uses main's gethostname() / env->getHostname() (the localHostname symbol feat: support multinode for json format output #54 relied on no longer exists on main).
  • Fixes the root cause: the sole recordDevices caller now passes deviceCount instead of worldSize, so device listings iterate the actual device count. PerfParserOutput reports worldSize under multinode (participant count), deviceCount otherwise.
  • Preserves main's runtime worldSize > 1 dispatch inside the #ifdef MULTINODE branch, so a MULTINODE build run with one rank still uses the single-node listing (avoids getPaddedProcessId(0) / log10(0)).
  • Header cleanup: recordDevices(int deviceCount); exchangeDeviceInfo declaration guarded by #ifdef MULTINODE.

Testing

Validated on real hardware (GB200/GB300 multinode) — the multinode --json path that previously failed with CUDA_ERROR_INVALID_DEVICE now completes and emits per-rank device info. Single-node and single-rank MULTINODE runs behave as before. Please re-run your CI / multinode validation to confirm on your side.

Known limitations (pre-existing on main, not addressed here)

These exist on main today and are out of scope for this rebase; flagging for visibility:

  • PerfParserOutput does not assign per-rank localDevice in multinode (device assignment is coupled to the text/JSON output helpers; the natural owner is MultiNodeEnv).
  • --pair UUID validation is per-rank and non-collective: a rank returning early on a missing UUID can leave peers blocked in MPI_Allgather.
  • The JSON warning field is now an array (from feat: support multinode for json format output #54) to retain multiple per-node warnings, a schema change from the previous string. Happy to gate this behind a compatibility decision if you prefer.

Co-authored-by: Shady Chan chenyulin.shady@bytedance.com

Support multinode when testing with json format output (`-j`),
which fixes these issues:
- Missing device info of remote nodes in json output
- Unexpected local node d2d bandwidth value due to variable
  `localDevice` not being updated through MPI
- MPI_ABORT invoked in multinode broadcast testcases and also in
  test with option `-j -p multinode`

Rebased NVIDIA#54 onto current main, resolving conflicts with the unified
multinode/single-node path and the `--pair`/UUID and per-node warning
logic added since v0.8. `recordDevices`' caller now passes `deviceCount`
instead of `worldSize`, so the JSON and perf-parser outputs iterate the
device count rather than the world size (the source of
CUDA_ERROR_INVALID_DEVICE on multinode `--json` runs).

Co-authored-by: Shady Chan <chenyulin.shady@bytedance.com>
Signed-off-by: Shady Chan <chenyulin.shady@bytedance.com>
Signed-off-by: jingxuz <jingxuz@nvidia.com>
@jingxu9x

Copy link
Copy Markdown
Author

@esitaridi can you help review this pr?

@jingxu9x jingxu9x closed this Sep 15, 2026
@jingxu9x
jingxu9x deleted the jingxuz/multinode-json-output branch September 15, 2026 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant