[None][feat] MiniMax-M3 NVFP4 low latency decode megakernel for Rubin (sm_107) TP2 / TP4 / TP8 in modeling_v2 by peihu-nv · Pull Request #20623 · NVIDIA/TensorRT-LLM

peihu-nv · 2026-10-19T19:16:44Z

Description

Depends on #20306 (flat staircase layout of the sm_103 targets). This PR adds only the last commit; please review that commit.

This PR brings the MiniMax-M3 NVFP4 decode targets to Rubin (sm_107): minimax_m3_nvfp4__sm_107__tp2, __tp4 and __tp8. They follow the flat layout of the sm_103 targets: one modeling.py per target, a weights.py weight table, and an ops/ catalog entry with certified cells and a test for every kernel the model calls. The router picks a target only for the matching checkpoint, SM and width, and modeling_v2: require makes it mandatory.

Performance

On Rubin (VR200), MiniMax-M3 NVFP4 AgentX with Eagle3, against the stock decode path (modeling_v2 off):

End to end: about 2.8× interactivity vs the best stock interactivity point (TP8 disaggregated, 1,112 tok/s/user, vs stock TP4 aggregated, 401).
Decode step: about 1.5× (TP2), 2.0× (TP4) and 2.4–2.7× (TP8) faster at batch 1.

sm_107 enablement

Catalog and routing: Arch.SM_107, and routing entries for (10, 7) at 2, 4 and 8 ranks, with the same drafter and runtime declines as sm_103.
Compiled kernels: minimaxM3Sm107Tp{2,4,8} with their trtllm::minimax_m3_sm107* thop ops, plus two shared GEN kernels: the TF32 tensor-core router GEMV and the spec-tail draft-input step.
CuTe DSL kernels: the decode MoE engines (1–8 rows, plus a 9–13-row engine at TP4), the MX GEMV, and the fused embedding + first RMSNorm.
Decode runtime: the step-input pack, skipped unchanged uploads, the spec step fold and the MSA decode metadata reuse. Each piece is inert until a target's RuntimeHooks turns it on, so other models are unchanged.
PrimTS (vendored FlashInfer task-scheduled attention): the dense layers and the drafter run its split-KV decode. The vendor patch scopes PrimTS's global WorkTileInfo rebind to its own compiles, so other CuTe DSL kernels (e.g. FA4) are unaffected.
KV fallback: the sm_107 NVFP4 → FP8 KV fallback skips MiniMax-M3 MSA, which keeps NVFP4 only on its sparse layers.
FMHA fallback: on SM107 the fallback FMHA serves paged prefill at the TRT-LLM Gen extra page sizes (the 128-token KV pages the MiniMax-M3 caches opt into) instead of refusing it; SM100 / SM103 keep the refusal.
Runtime items (target opt-in): the pinned FlashInfer CuTe DSL MXFP8 tactics above 32 rows on SM107, the post-warmup late-compile detector, and first-content emission (a request's first content after two iterations) are armed only when the model's target opts in through its RuntimeHooks; the MiniMax-M3 sm_107 targets do, and other models are unchanged. The AllReduce tuner leaves NCCL_SYMMETRIC out of its candidates only with TRTLLM_AR_TUNE_NCCL_SYMMETRIC=0 (opt-in; the default keeps the old candidates).

The targets

All three share one structure; they differ in rank geometry and row ranges.

	TP2	TP4	TP8
Decode step bound	192 rows (fast paths to 128)	48 rows	8 rows
MoE engine	1–8 rows (Triton finalize from 49)	1–8 + 9–13 rows	1–8 rows
Sparse decode attention	`m3_sparse_decode_lc`	`m3_sparse_decode_lc`	`m3_sparse_decode_lc8`

Boundaries: multicast all-reduce fused with the next RMSNorm and with the MoE finalize.
GEMMs: an MX GEMV for low-row MXFP8 projections and drafter GEMMs; at TP2 the LM head is split across the two dies.
Speculation (Eagle3, three drafts): a fused draft step and the target's own sampled accept. At TP4 the sampled verify runs two rounds (single-round accept off), matching the deployed configuration.
No tuning knobs: the targets read no tuning knobs from the environment; the only variable is the opt-in runtime-value dump diagnostic (TRTLLM_MODELING_V2_VALUE_DUMP), as in the sm_103 targets. Every fast path is on by default and has one module constant that turns it off. Capacity bounds carry a # capacity: annotation.

Aggregated (wide) decode at TP2

Rows: the TP2 target serves aggregated decode steps of up to 48 requests (1..192 rows, each request verifying 1 + 3 drafts). Fast-path cells cover up to 128 rows; 129..192-row steps take the stock-exact paths.
Wide cells: steps above 32 rows run the generic layer with 48-row multicast boundaries and MC finalize, the tensor-core router (9..48 rows), the Triton MoE finalize (49+ rows) and the 33..48-row GEMVs.
Mixed steps: steps with context rows run under breakable prefill CUDA graphs, with the drafter's attention in an eager break.

Numerics. No lossy steps. Most kernels are bitwise equal to the stock op they replace; the rest reduce in a fixed order, are deterministic in eager and graph mode, and stay within the stock kernel's error band. One known divergence: the TP4 drafter all-reduce + RMSNorm always sums like ONESHOT, so under a forced NCCL strategy its rounding differs from stock NCCL. Against the earlier wide TP2 reference, target tokens are identical: at 12+ concurrent requests under Eagle3 some draft tokens can resolve near-ties differently (target logits bitwise), and without speculation (not a production configuration) decode steps of 9..48 rows are bitwise on the first such step and differ after it, with identical output tokens. Steps of 1..8 rows are bitwise for prompts up to about 12K tokens, and at TP8 also for a single 83K-token request (prefill and decode).

Not in this PR: batch 4/8 decode at TP4 / TP8 (follow-up). The prefill (context) optimizations are in #20629.

Known behaviours: The pinned SM107 MXFP8 GEMM tactics are armed process-wide by an executor whose model opts in, so another model in the same process at the same GEMM shape also uses them; its outputs are bitwise unchanged (every pinned tactic equals the default), and the LLM API runs one executor per worker process.

Test Coverage

Bitwise A/B vs the pre-restructure sm_107 targets (TP2/TP4/TP8, no speculation and Eagle3, every rank, prompts up to about 12K tokens; at TP8 also one 83K-token request): identical tokens and per-step output hashes. The only exception is the unread tail of new_tokens in Eagle3 context steps. Re-run on the final commit at TP2 and TP4 (no speculation and Eagle3): bitwise at every step of 1..8 rows; at TP2 the first difference is a 45-row context step, which this PR moves to the wide cells.
Wide TP2 vs the aggregated (wide) reference target at 9..128 rows, Eagle3 and no speculation: target tokens identical in every phase and logits bitwise on every recorded step under Eagle3; the two deltas above are the only differences.
FMHA fallback at 128-token pages on SM107: a GPU oracle (fresh contexts beside generation rows, cached prefixes intact, K/V stored at their slots) and a CPU gate, all cases in band.
Unit tests on Rubin, one session per file: TP2 1449, TP4 1659, TP8 1415 tests, all passing. Each target's drafter context cell is a strict expected failure because the SM107 non-paged context FMHA does not apply RoPE (shared runtime issue, tracked separately).
Decode runtime tests (executor/, speculative/, attention): 255 pass, 1 skipped.
Delivery and capacity checkers: TP2 99/99, TP4 107/107, TP8 97/97.
Accuracy at TP2 vs the pre-restructure targets: GSM8K (1319) McNemar p = 0.30 greedy, p = 1.00 sampled; the Eagle3 natural acceptance length over 264 long-context requests is identical. At TP4 and TP8, GSM8K (1319) vs the pre-restructure targets: TP4 p = 1.00 greedy, p = 1.00 sampled; TP8 p = 0.38 greedy, p = 0.29 sampled.
Decode perf vs the pre-restructure targets, same node, both run orders: TP2, TP4 and TP8 within noise (TP8 compared at equal decode work: the synthetic runs' generated text, and so the number of distinct routed experts per step, varies from server start to server start).

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

peihu-nv · 2026-10-19T19:34:03Z

/bot run --disable-fail-fast

peihu-nv · 2026-10-20T02:27:42Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-20T02:35:50Z

PR_Github #79103 [ run ] triggered by Bot. Commit: 5d24e81 Link to invocation

tensorrt-cicd · 2026-10-20T07:39:28Z

PR_Github #79103 [ run ] completed with state FAILURE. Commit: 5d24e81
/LLM/main/L0_MergeRequest_PR pipeline #65162 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

peihu-nv · 2026-10-20T12:18:03Z

/bot run --disable-fail-fast

peihu-nv · 2026-10-20T15:22:21Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-20T15:31:17Z

PR_Github #79291 [ run ] triggered by Bot. Commit: 7333ac8 Link to invocation

tensorrt-cicd · 2026-10-20T20:21:49Z

PR_Github #79291 [ run ] completed with state SUCCESS. Commit: 7333ac8
/LLM/main/L0_MergeRequest_PR pipeline #65314 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

peihu-nv · 2026-10-20T20:31:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-20T20:39:59Z

PR_Github #79385 [ run ] triggered by Bot. Commit: 8b49063 Link to invocation

tensorrt-cicd · 2026-10-21T01:03:12Z

PR_Github #79385 [ run ] completed with state SUCCESS. Commit: 8b49063
/LLM/main/L0_MergeRequest_PR pipeline #65380 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

peihu-nv · 2026-10-21T11:15:47Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-21T11:22:56Z

PR_Github #79616 [ run ] triggered by Bot. Commit: 8b49063 Link to invocation

peihu-nv · 2026-10-21T22:07:41Z

/bot run --disable-fail-fast

Brings in NVIDIA#20623 at ffa76ab (the NVIDIA#20306 head 300c1c1 merged in, sm107 target kernels under ops). No change to the context path. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

tensorrt-cicd · 2026-10-21T22:15:12Z

PR_Github #79695 [ run ] triggered by Bot. Commit: ffa76ab Link to invocation

tensorrt-cicd · 2026-10-21T22:18:19Z

PR_Github #79616 [ run ] completed with state ABORTED. Commit: 8b49063

Link to invocation

peihu-nv · 2026-10-31T20:47:20Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-31T20:54:21Z

PR_Github #81809 [ run ] triggered by Bot. Commit: 50074ec Link to invocation

tensorrt-cicd · 2026-11-01T01:38:06Z

PR_Github #81809 [ run ] completed with state SUCCESS. Commit: 50074ec
/LLM/main/L0_MergeRequest_PR pipeline #67464 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

…gets Move common Python operators, kernel implementations, native code and test helpers to the MiniMax-M3 family while preserving TP-specific assembly, shape/tactic tables, topology and runtime state. Keep existing numerical and dispatch policies, and add shared distributed sampling coverage. Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

peihu-nv · 2026-11-01T20:46:20Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-01T20:53:48Z

PR_Github #81870 [ run ] triggered by Bot. Commit: 04f4a3f Link to invocation

tensorrt-cicd · 2026-11-02T00:52:26Z

PR_Github #81870 [ run ] completed with state SUCCESS. Commit: 04f4a3f
/LLM/main/L0_MergeRequest_PR pipeline #67522 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

peihu-nv · 2026-11-02T06:04:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-02T06:11:10Z

PR_Github #81918 [ run ] triggered by Bot. Commit: 61239b7 Link to invocation

tensorrt-cicd · 2026-11-02T09:57:22Z

PR_Github #81918 [ run ] completed with state SUCCESS. Commit: 61239b7
/LLM/main/L0_MergeRequest_PR pipeline #67565 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

peihu-nv · 2026-11-02T10:46:13Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-02T10:52:36Z

PR_Github #82000 [ run ] triggered by Bot. Commit: 23c0805 Link to invocation

tensorrt-cicd · 2026-11-02T23:17:02Z

PR_Github #82000 [ run ] completed with state FAILURE. Commit: 23c0805
/LLM/main/L0_MergeRequest_PR pipeline #67640 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

peihu-nv · 2026-11-02T23:39:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-02T23:47:00Z

PR_Github #82111 [ run ] triggered by Bot. Commit: 23c0805 Link to invocation

tensorrt-cicd · 2026-11-03T00:37:24Z

PR_Github #82111 [ run ] completed with state SUCCESS. Commit: 23c0805
/LLM/main/L0_MergeRequest_PR pipeline #67744 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

github-actions Bot assigned peihu-nv Oct 19, 2026

peihu-nv changed the title ~~[None][feat] MiniMax-M3 NVFP4 decode targets for Rubin (sm_107) TP2 / TP4 / TP8 in modeling_v2~~ [None][feat] MiniMax-M3 NVFP4 low latency decode megakernel for Rubin (sm_107) TP2 / TP4 / TP8 in modeling_v2 Oct 19, 2026

peihu-nv mentioned this pull request Oct 19, 2026

[None][perf] Optimize MiniMax-M3 NVFP4 context (prefill) path on Rubin (SM107) #20629

Open

1 task done

peihu-nv force-pushed the peihengh/m3-vr-sm107-decode branch from cd5f671 to 5d24e81 Compare October 20, 2026 02:27

peihu-nv force-pushed the peihengh/m3-vr-sm107-decode branch from 5d24e81 to 4d16da2 Compare October 20, 2026 12:17

peihu-nv force-pushed the peihengh/m3-vr-sm107-decode branch from 4d16da2 to 7333ac8 Compare October 20, 2026 15:21

peihu-nv marked this pull request as ready for review October 21, 2026 23:49

peihu-nv requested review from a team as code owners October 21, 2026 23:49

yuxianq reviewed Oct 29, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/attention/backends/trtllm.py Resolved

yuxianq reviewed Oct 29, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/attention/backends/fmha/fallback.py Resolved

peihu-nv force-pushed the peihengh/m3-vr-sm107-decode branch from 646c6f4 to 50074ec Compare October 31, 2026 20:47

peihu-nv added 6 commits November 1, 2026 12:44

[None][feat] Add MiniMax-M3 Rubin decode targets

4a16b99

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

[None][fix] Align Rubin tests with rebased runtime and CI routing

fc05ff3

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

[None][fix] Drop unrelated GLM import workarounds

9aeee2c

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

[None][fix] Address Rubin kernel and runtime review findings

ef86ded

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

[None][test] Cover strided Rubin multi-query attention inputs

950592f

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

peihu-nv force-pushed the peihengh/m3-vr-sm107-decode branch from 50074ec to 04f4a3f Compare November 1, 2026 20:45

leslie-fang25 approved these changes Nov 2, 2026

View reviewed changes

[None][test] Fix shared M3 CI import and fixture contracts

61239b7

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

[None][test] Share TP8 sampling fixtures outside test modules

23c0805

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

greg-kwasniewski1 mentioned this pull request Nov 2, 2026

[None][perf] Add a small-M MXFP8 quantization kernel for swizzled scale factors #21444

Draft

1 task

peihu-nv enabled auto-merge (squash) November 3, 2026 00:29

Conversation

peihu-nv commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Performance

sm_107 enablement

The targets

Aggregated (wide) decode at TP2

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

peihu-nv commented Oct 19, 2026

Uh oh!

peihu-nv commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

peihu-nv commented Oct 20, 2026

Uh oh!

peihu-nv commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

peihu-nv commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 21, 2026

Uh oh!

peihu-nv commented Oct 21, 2026

Uh oh!

tensorrt-cicd commented Oct 21, 2026

Uh oh!

peihu-nv commented Oct 21, 2026

Uh oh!

tensorrt-cicd commented Oct 21, 2026

Uh oh!

tensorrt-cicd commented Oct 21, 2026

Uh oh!

Uh oh!

Uh oh!

peihu-nv commented Oct 31, 2026

Uh oh!

tensorrt-cicd commented Oct 31, 2026

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

peihu-nv commented Nov 1, 2026

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

tensorrt-cicd commented Nov 2, 2026

Uh oh!

peihu-nv commented Nov 2, 2026

Uh oh!

tensorrt-cicd commented Nov 2, 2026

Uh oh!

tensorrt-cicd commented Nov 2, 2026

Uh oh!

peihu-nv commented Nov 2, 2026

Uh oh!

tensorrt-cicd commented Nov 2, 2026

Uh oh!

tensorrt-cicd commented Nov 2, 2026

Uh oh!

peihu-nv commented Nov 2, 2026

Uh oh!

tensorrt-cicd commented Nov 2, 2026

Uh oh!

tensorrt-cicd commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

peihu-nv commented Oct 19, 2026 •

edited

Loading