[chzblych]

Continued in NVIDIA/TensorRT-LLM#21516 using the same source branch and commit cd59d75d1ea0b91cdb16091e5d91c9d3af21bd9c.

Description

Problem

Rocky Linux 8's hwloc 2.2.0 is too old for the NIXL LIBFABRIC build. Installing the fallback under /usr/local and preferring it globally changes the hwloc selected by Open MPI and other consumers.

A successful container build also does not establish that KV cache V2 transfers work through the NIXL LIBFABRIC backend on EFA hardware.

Solution

• Install fallback hwloc 2.12.2 under /opt/nvidia/nixl-deps/hwloc and select it for the NIXL build through PKG_CONFIG_PATH. Preserve suitable existing HPC-X or system hwloc installations and leave global runtime library search configuration unchanged. Runtime use of NIXL LIBFABRIC still requires a compatible hwloc library.
• Add a two-node B300 functional CI case using the existing disaggregated launcher and standard Slurm segment handling. Pin its CI placement to the EFA-capable backend. One context worker and one generation worker use one GPU each, KV cache V2 and the NIXL LIBFABRIC backend.
• Require eight long-prompt requests at concurrency four to match ordinary greedy inference. Check actual worker EFA activity and match nonzero send bytes to receiver completions by request ID. Missing EFA support fails the test; there is no throughput or latency threshold.
• Register the test and its explicit dependencies for multi-GPU CI selection, retain CPU checks against false-positive transport evidence, update the hwloc documentation, and complete license notices in the touched installers and tests.

Test Coverage

Validation

Reviewed head: cd59d75d1ea0b91cdb16091e5d91c9d3af21bd9c, based on upstream main ac0c1fda181e5d5cd6f5909d391e522035b4c816.

• Current source checks: 111 CPU checks passed, along with Groovy placement/dependency-trigger checks, stage mapping, shell syntax and applicable pre-commit checks. All five follow-up commits retain DCO sign-offs.
• Earlier isolated-hwloc validation: clean Rocky8 devel builds for Python 3.10 and 3.12, full SM90 wheel build/install/import, and MPI/NIXL UCX initialization checks passed. Process mappings retained the system hwloc. These builds were not rerun after this rebase.
• Earlier B300 validation: the complete two-node case passed both without --segment and with --segment=2. Both runs completed eight matching inference results and eight nonzero EFA KV transfers; all harness roles and Slurm steps passed. Fourteen evidence-validator unit cases also passed inside the image.
• Those B300 runs used test source c097a6f29ec28c9d516a27818599efe7b2b54796, CI wheel source b2ade286de557b8dec22d81d6f7bb1e244e2f255, and the formal NGC PyTorch 26.09 image at platform digest sha256:6435f9aaa7e0a6e0b37047d031b5300fb7e733759895541df99c93e09d18b6eb. They used NIXL 1.5.2 and AWS libfabric 2.7.0amzn2.0 without NIXL_DISABLE_CUDA_ADDR_WA, through the repository's Slurm launcher with task-local wheel/dependency setup.

Full Jenkins L0 and fresh image/GPU validation remain pending for the rebased head.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR follows TRT-LLM Coding Guidelines to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions).

• If PR introduces API changes, an appropriate PR label is added: api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities.

• CODEOWNERS updated if ownership changes.

• Documentation updated as needed.

• Update the architecture diagram if there is a significant design change.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.