[None][feat] Muse Spark Open modeling_v2 TEP target: a per-deployment lever resolver and the Vera Rubin high-concurrency, mid-concurrency and low-latency fast paths by xwang233 · Pull Request #21377 · NVIDIA/TensorRT-LLM

xwang233 · 2026-10-31T21:26:50Z

Description

This PR adds a per-deployment lever resolver to the Muse Spark Open modeling_v2 TEP target, and with it the Vera Rubin (sm107) fast paths for the high-concurrency, mid-concurrency (TEP4 decode steps of 2–4 requests) and low-latency (one request) regions. One build then serves the whole agentic Pareto curve: aggregated TEP8 / TEP4 and the context and generation servers of disaggregated deployments. It follows #21143. The changes are 12 commits:

7 inside the Staircase target (_experimental/modeling_v2/models/musespark_open/...__tep): the resolver, five groups of fast paths, and the high-concurrency selection;
1 accuracy fix inside the target: router per row only on generation servers;
4 opt-in shared-path changes: KV cache manager v2, cold-page KV compression, disaggregated serving, and a PyExecutor runtime hook. One more commit adds an opt-in model option (NVFP4 experts for layers 0 and 61).

The lever resolver (<target>/levers.py).

dispatch.compose resolves one table of levers once per process from the deployment it sees: device, TP width, max_batch_size, max_num_tokens, the serving role (aggregated, or the context / generation side of a disaggregated server) and the drafter's context arena.
Composition and bitwise levers follow their rule; numeric levers also need their accuracy gate's deployment class; lossy levers are never on unless requested.
Every lever has a knob, MUSESPARK_OPEN_LEVER_<NAME>=auto|on|off, plus declared thresholds …__<PARAM>, MUSESPARK_OPEN_LEVER__ROLE and MUSESPARK_OPEN_LEVER__DEFAULT=baseline. baseline selects exactly what the target served before this change, as a control.
Bad names or values, requests a structural rule excludes, and rank disagreement raise before anything is installed. The installed table is logged once as JSON, and per-step declines are counted.
On sm103 (GB300) every lever resolves as the previous selection; a host test pins this.

Region selections on sm107 under auto:

TEP8, one request: one-stream step loop on the execution stream, incremental block offsets, cooperative head-sampler fold, and on a dense drafter arena the drafter K/V side chain, early fc fork and commit-written preamble.
TEP4 decode steps of 2–4 requests (16–32 verify rows):
- wide direct pool / unpool transports with the half expert window;
- o_proj published directly into the wide post-attention hop;
- finalize transport for 17–32 tokens;
- on generation servers only, router per row (each row routes with its own one-request bits).
TEP4 with max_batch_size 6–47:
- packed multi-request prep append, bitwise as the per-sequence grid;
- parallel windowed context commit from 5 requests (same bytes);
- the wide paths and the cooperative fold.
Aggregated TEP4 with max_batch_size ≥ 48: the high-concurrency selection. That is packed prep, parallel commit from 1 request and the graph-start QKV prefetch, with the wide paths and the cooperative fold off.
Opt-in for large context calls (lossy): FP8 copies of the qkv / o_proj / pool / unpool projections for context calls of ≥ 1024 rows, with an MXFP8 attention-input gather. It is off unless requested; only the high-concurrency recipe turns it on.

Shared paths: opt-in, off by default. With the variables or arguments unset, behaviour is unchanged.

KVCacheManagerV2, each switch independent:
- TLLM_KV_CACHE_MANAGER_V2_STREAM_ORDERED_EVENTS=1: skips event waits the stream order implies;
- …_KEY_CHAIN_MEMO=1: memoises a prompt's block-key chain;
- …_LAZY_SLOT_SCRUB=1: scrubs recycled slots once per batch allocation.
ColdPageQuantizationCompressionConfig gains global_scale (identity default, fp8_block_amax) and uncompressed_layer_types. DFlash without a separate draft KV cache now passes the compression check. The llm_args golden manifest records the two fields.
Disaggregated serving:
- TRTLLM_DISAGG_GEN_PREFETCH_REQUESTS=N admits up to N generation-only requests beyond max_num_active_requests so their KV transfer overlaps the running decode; the forward batch stays capped at max_batch_size.
- LoadBalancingRouter gains max_inflight_per_server / hold_timeout_s.
RuntimeHooks.step_loop_on_execution_stream (False by default): a model can declare it to run the overlap loop's sampler and input preparation on the execution stream.
MUSESPARK_OPEN_L061_EXPERTS_NVFP4=1 builds the experts of the quantization-excluded MoE layers 0 and 61 as NVFP4 from BF16 at load. This is a precision change, never enabled implicitly.

Measured gains. AgentX traces on the 1M-token corpus, Vera Rubin, full protocol, forced draft acceptance 3.05. x = p10 per-user output throughput; y = total throughput per GPU. Each row compares this tree with, in the same wave:

the code of [None][feat] Muse Spark Open modeling_v2 TEP target: GB300 and Vera Rubin fast paths, and the DFlash 2 drafter transform on the sharded draft head #21143 (its delivery build);
this tree with MUSESPARK_OPEN_LEVER__DEFAULT=baseline, i.e. the target's previous selection.

Point	vs #21143 code Δx / Δy	vs previous selection Δx / Δy
TEP8 c1	+5.5 % / +1.1 %	+5.0 % / +0.3 %
TEP4 c32, `max_batch_size` 4	+9.1 % / +5.8 %	+4.8 % / +3.1 %
TEP4 c32, `max_batch_size` 8	+5.0 % / +1.8 %	−0.1 % / +0.0 %
TEP4 c48, `max_batch_size` 32	+10.0 % / +3.0 %	+1.3 % / +0.0 %

High-concurrency point (TEP4 c96, max_batch_size 48, with the high-concurrency recipe and its context-FP8 opt-in). Against the earlier high-concurrency build of this target, three high-mode runs of this tree give +0.1 % x / +0.3 % y. The resolver's earlier choice at this size (wide paths and cooperative fold on) was −2.2 % y, which is why the high-concurrency selection is the default from max_batch_size 48.
⚠️ c96 low mode. 1 of 4 runs of the final tree landed in a low mode, with TTFT p90 about 40 % higher and x about 6 % lower. That mode also appears with the previous selection, and its cause is not identified. Over all four runs: −1.4 % x / −0.3 % y.
Other recipes. The remaining recipes on the curve were measured once each against their previous published results (different days): +2.6 to +8.4 % x and +0.5 to +2.5 % y on aggregated TEP4 (c24 … c64). The disaggregated mid-concurrency points reproduce their earlier builds within −0.2 to +2.0 % x.
Every run installed exactly its expected lever table.

Accuracy. Spec decoding on with natural acceptance; long-preamble GSM8K, GPQA-Diamond and a prefix-reuse probe, greedy; pooled paired McNemar.

The high-concurrency point against its earlier build, TEP8 c1 against the previous selection, and a 1 ‖ 1 TEP4 B2 generation server against the previous selection (GSM8K) all pass (p ≥ 0.14; acceptance-length ratio within ±1 %).
Router per row lost GPQA on aggregated TEP4 servers of max_batch_size 2–4: −4.0 pp, p 0.002. With the lever off, the same check passes (−0.25 pp, p 1.0). On a generation server it costs nothing (+1.0 pp, p 0.50).
So its gate holds on generation servers only (commit [None][fix] … router per row only on generation servers).

Validation.

The tree was built (wheel for sm103 and sm107) and measured on feat/musespark-open-agentx at c55183f445.
GPU checks ran on sm107 at TP4 and TP8:
- the upstream target tests and the lever tests;
- the bitwise checks of the new kernels against their predecessors (packed prep vs grid prep; windowed commit one-program vs parallel vs graph replay; cooperative fold vs the plain fold over rows 1..128 and the 129-row fallback).
This branch is that tree replayed onto the current tip 51daa26acb ([None][fix] Muse modeling_v2: half-window early expert ids on Vera Rubin, and context_kv_fork under KV cache estimation #21364). One conflict, in ops/comm/pool_direct_route.py, is resolved by keeping both sides: the wide path's CTA-relative shared index and [None][fix] Muse modeling_v2: half-window early expert ids on Vera Rubin, and context_kv_fork under KV cache estimation #21364's place_half. The resulting tree equals the measured tree plus [None][fix] Muse modeling_v2: half-window early expert ids on Vera Rubin, and context_kv_fork under KV cache estimation #21364's two hunks exactly. The replayed tree was not rebuilt.
With every knob unset, the shared paths are unchanged. The DFlash drafter stays on VANILLA in the recipes; the TRTLLM drafter backend is not measured on Vera Rubin.

Test Coverage

tests/unittest/_torch/modeling_v2/models/musespark_open/.../test_modeling_v2_levers.py (new, CPU-only): the knob grammar, overrides, structural rules, baseline reproduction, and every region's lever table, the high-concurrency selection and router-per-row gates included. 180 tests. It is in no CI test list yet; the candidate is l0_cpu.
.../test_modeling_v2_drafter_context.py: the plan over a paged drafter arena, whose only structural exclusion is the window compaction.
tests/unittest/kv_cache_manager_v2_tests/test_stream_ordered_events.py, test_key_chain_memo.py; tests/unittest/_torch/executor/kv_cache/test_kvcm2_integration.py: the KV cache manager v2 switches.
tests/unittest/_torch/kv_cache_compression/test_quantization_for_cold_page.py, test_nvfp4_cold_page_fp8_global_scale_gpu.py; tests/unittest/llmapi/test_llm_args.py: the cold-page fields.
tests/unittest/_torch/executor/test_disagg_gen_prefetch.py, tests/unittest/disaggregated/test_router_hold.py: generation prefetch admission and router hold.
tests/unittest/_torch/executor/test_runtime_hooks.py: the execution-stream step loop hook.
tests/unittest/_torch/modeling/test_musespark_open_l061_nvfp4.py, test_musespark_open_context_fp8_dense.py: the NVFP4 layer-0/61 experts and the context FP8 dense projections.
accuracy/modeling_v2/test_musespark_open.py (on this branch): GSM8K and mean acceptance length under modeling_v2="require".

PR Checklist

Please check this after reviewing the above items as appropriate for this PR.

🤖 Generated with Claude Code

…y chain memo and lazy slot scrub Three independent changes to the KV cache manager v2 admission path. Each is off by default; with its variable unset the manager behaves as before. - TLLM_KV_CACHE_MANAGER_V2_STREAM_ORDERED_EVENTS=1: page resume and close skip event waits and merges that the stream order already implies, and wait once per distinct event instead of once per page. - TLLM_KV_CACHE_MANAGER_V2_KEY_CHAIN_MEMO=1: the radix tree memoises the block-key chain of a token sequence, so the reuse probe, the first-block probe and the cache creation of one request hash its prompt once. - TLLM_KV_CACHE_MANAGER_V2_LAZY_SLOT_SCRUB=1: a slot allocator scrubs the events of recycled slots once per batch allocation instead of polling them per slot. The nanobind module exposes the switches and the event counters for the unit tests. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…ressed layer types; DFlash without a separate draft KV cache ColdPageQuantizationCompressionConfig gains two fields, both defaulting to the previous behaviour: - global_scale: identity (default) or fp8_block_amax. For an FP8 KV cache without a ModelOpt scale, fp8_block_amax uses a global scale of 6, which maps each block's E4M3 absolute maximum onto an exact E4M3 block scale. - uncompressed_layer_types: layer types (sliding_window, full_attention) whose pages stay uncompressed in the cold tier. The KV cache compression check now also admits DFlash when it has no separate draft KV cache: its drafter keeps the context K/V in worker-owned buffers, so the codec only sees target pages. The llm_args golden manifest records the two new fields. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…ission and router held placement Generation workers: - TRTLLM_DISAGG_GEN_PREFETCH_REQUESTS=N (0 by default) admits up to N generation-only requests beyond max_num_active_requests, so their KV transfer overlaps the running decode. The forward batch stays capped at max_batch_size. The engine widens its sequence-slot pool and the KV cache manager v2 its lease budget by the same N. It is off under TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN, attention DP and pipeline parallelism. Requests that do not fit go back to the front of the waiting queue, which keeps FCFS order. Router: - LoadBalancingRouter gains max_inflight_per_server and hold_timeout_s. With a cap set, a request is placed only on a server below the cap, and otherwise held until one frees up; a request held longer than the timeout falls back to the least-loaded server. Without a cap, placement is unchanged. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…ap loop on the execution stream RuntimeHooks gains step_loop_on_execution_stream (False by default). When a model declares it, the overlap loop's thread makes the execution stream current, so the sampler and the next step's input preparation queue behind the forward on one stream instead of crossing streams twice per step. Models that do not declare it are unchanged. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…on-excluded MoE layers The checkpoint keeps the experts of its quantization-excluded MoE layers (0 and 61) in BF16. With MUSESPARK_OPEN_L061_EXPERTS_NVFP4=1 (off by default) those experts are built on the global NVFP4 leaf and quantized from BF16 at load. MUSESPARK_OPEN_L061_EXPERTS_NVFP4_LAYERS selects a subset of the layers, and MUSESPARK_OPEN_L061_EXPERTS_NVFP4_FC2_AMAX supplies the FC2 input amax per layer. This is a precision change and is never enabled implicitly. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…projections for large context calls New kernels/context_fp8_dense: at load, FP8 copies of the qkv, o_proj, pool and unpool projection weights; at run time, context calls of at least min_rows rows (1024 and above) quantize their input with a one-pass MXFP8 quant and run the FP8 GEMM with fixed Rubin tactics, plus an optional MXFP8 attention-input gather on sequence-parallel steps. Decode steps never reach the row floor. The install refuses unless the device is sm107, the Rubin CuTe DSL is available and the projections are plain bias-free BF16. This is a precision change. The lever resolver keeps it off unless it is requested explicitly. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…p append and a parallel windowed context commit Decode steps of several requests on sm107 TEP4: - multi_request_prep_append runs the packed-geometry kernel for calls of at least packed_min_rows rows (one program per head over all sequences) instead of the per-sequence grid; the result is bitwise the same. - context_kv_commit_windowed commits the drafter's windowed context arena with one program per request and a tuned compaction for steps of at least parallel_min_requests requests; smaller steps keep the one-program commit. Both write the same bytes. - hot_replicas skips experts that were quantized at load, which have no census table. The thresholds are arguments set by the lever resolver. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…or decode steps of 16 to 32 rows Decode steps of two to four requests (16 to 32 verify rows) on sm107 TEP4: - wide_prologue routes each row with its own one-request bits on the wide router (router shard rows), so a row routes as it would alone; - pool_into_route and unpool_into_boundary run the direct pool and unpool transports with the half expert window at 16, 24 and 32 rows; - attention_output_into_hop publishes o_proj directly into the wide post-attention hop transport at 16, 24 and 32 rows; - finalize_transport serves finalize calls of 17 to 32 tokens. The 8-row layout shares the kernel builders; its arithmetic is unchanged. Which of these paths a deployment serves is decided by the lever resolver. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…et chain and a cooperative head-sampler fold Paths for decode steps of one or a few requests on sm107: - drafter K/V side chain: on a dense full-length drafter arena, the context K/V of the step is projected and stored on a side stream forked after the last capture layer, with the drafter's fixed GEMM tactics; the fc projection can fork early, and the commit kernel also writes the drafter forward's integer preamble (context_kv_store, context_kv_commit, context_kv_fork, verify_fold, drafter_block_decode). - head sampler: a cooperative kernel A computes the per-row states of calls of up to 128 rows with the same per-element cap and combine order as the grid kernel, so states and tokens are bitwise the same; wider calls keep the grid kernel. - the target head can serve without the lm_head inbox, and can gather its logits with a multimem all-gather when every rank builds the symmetric buffer. - runtime: the block-offset stager copies only the runs of page indices that changed since the previous step, keyed by value, and resets on storage, scale or offset changes. Each path declines per step when its arena or shape does not apply. Which of them a deployment serves is decided by the lever resolver. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…cks the fast paths per deployment New levers.py: one table of levers with a rule each. dispatch.compose resolves it once per process from the routing key and the deployment it sees: device, tensor-parallel width, max_batch_size, max_num_tokens, the serving role (aggregated, or the context / generation side of a disaggregated server) and the drafter's context arena (dense, windowed or paged). Installers read their decision and thresholds from the plan; no other module of the target reads the environment. - Lever kinds: composition and bitwise levers follow their rule; numeric levers also need their deployment class; lossy levers are never on unless requested; flagged levers stay off until requested. - Knobs: MUSESPARK_OPEN_LEVER_<NAME>=auto|on|off per lever, declared parameters MUSESPARK_OPEN_LEVER_<NAME>__<PARAM>, MUSESPARK_OPEN_LEVER__ROLE and MUSESPARK_OPEN_LEVER__DEFAULT=auto|baseline. Unset means auto. baseline selects the levers the target served before this change, as a control. Unknown names, bad values, out-of-range parameters, a request a structural rule excludes and disagreement across ranks raise before anything is installed. - Structural rules: the drafter arena selects the commit and side-chain paths; TRTLLM_KV_DEFER_CLOSE=1 excludes the execution-stream loop; UCX_RNDV_THRESH may not be set on a disaggregated role. - The resolved and the installed lever tables are logged once on rank 0, the installed one as JSON after the last install hook, and per-step declines are counted. - The modeling_v2 router records the engine's deployment facts per target class. Host-only tests cover the grammar, the overrides, the structural rules, baseline reproduction and the per-deployment lever tables. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…neration servers wide_router_per_row is a lossless-numeric lever: each verify row routes with its own one-request bits, so near-tie experts can resolve differently from the built-in wide router. Its accuracy gate now holds only on generation servers at TEP4 with max_batch_size 2..4; aggregated servers of the same batch range resolve it off under auto (source "gate"). The lever tests now expect it off at agg mbs 2..4 and check that an aggregated server built from a generation config differs from that generation server only in this lever. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…election Aggregated sm107 TEP4 servers with max_batch_size of at least HC_MIN_BATCH (48) resolve under auto to the high-concurrency selection: the wide decode transports, the wide o_proj publish, the wide finalize transport and the cooperative head-sampler fold are off; the graph-start QKV prefetch is on; and the parallel windowed context commit starts from one request. A lever Param can now declare a per-deployment default (default_for); a knob still overrides it. Generation servers, TEP8 and smaller batches keep their selection. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Tabrizian · 2026-11-01T00:37:12Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-01T00:44:55Z

PR_Github #81813 [ run ] triggered by Bot. Commit: 110d5c2 Link to invocation

tensorrt-cicd · 2026-11-01T03:14:57Z

PR_Github #81813 [ run ] completed with state SUCCESS. Commit: 110d5c2
/LLM/main/L0_MergeRequest_PR pipeline #67468 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Tabrizian · 2026-11-01T05:12:09Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-01T05:18:54Z

PR_Github #81824 [ run ] triggered by Bot. Commit: 110d5c2 Link to invocation

tensorrt-cicd · 2026-11-01T06:39:36Z

PR_Github #81824 [ run ] completed with state FAILURE. Commit: 110d5c2
/LLM/main/L0_MergeRequest_PR pipeline #67477 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

feat/musespark-open-agentx's aarch64 build configs and docker CUDA_ARCHS predate main's sm_107 addition, so the SBSA wheel and release image carry no Vera Rubin code, while this PR's TEP fast paths target Vera Rubin. Add 107-real to both aarch64 WHEEL_ARCHS and to the arm64 docker CUDA_ARCHS, so a side-branch release of this branch runs on Vera Rubin as well as GB300. Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>

Tabrizian · 2026-11-02T16:15:19Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-02T16:24:15Z

PR_Github #82033 [ run ] triggered by Bot. Commit: 13b0b68 Link to invocation

tensorrt-cicd · 2026-11-02T19:10:30Z

PR_Github #82033 [ run ] completed with state FAILURE. Commit: 13b0b68
/LLM/main/L0_MergeRequest_PR pipeline #67670 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

… for bandit bandit's B105 reads PRIORITIZE_FIRST_TOKEN_ENV = "TLLM_DISAGG_GEN_PRIORITIZE_FIRST_TOKEN" as a hardcoded password (the value ends in TOKEN) and fails the Release-Check and the pre-commit workflow's bandit step. It is an environment variable's name: mark the line # nosec B105, as the other token-named constants are. Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>

…ters PLCScanningSetup renamed runContainerScanning to runContainerOssScanning and added runContainerMalwareScanning (main: PLC Pipeline update, NVIDIA#19557). This branch's L0_MergeRequest.groovy predates that and still passes runContainerScanning=false, which the job ignores: a merge request that touches an OSS-compliance file (here tensorrt_llm/usage/llm_args_golden_manifest.json) ran the container OSS scan at its default, looked for a post-merge image of the PR commit and failed [OSS-Compliance-Check]. Pass the job's current parameter names, with the values main passes, in both PLC calls. Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com>

Tabrizian · 2026-11-02T23:53:24Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-03T00:01:06Z

PR_Github #82117 [ run ] triggered by Bot. Commit: e05bfc4 Link to invocation

tensorrt-cicd · 2026-11-03T01:36:02Z

PR_Github #82117 [ run ] completed with state SUCCESS. Commit: e05bfc4
/LLM/main/L0_MergeRequest_PR pipeline #67750 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Tabrizian · 2026-11-03T02:33:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-03T02:41:37Z

PR_Github #82156 [ run ] triggered by Bot. Commit: e05bfc4 Link to invocation

tensorrt-cicd · 2026-11-03T04:54:15Z

PR_Github #82156 [ run ] completed with state FAILURE. Commit: e05bfc4
/LLM/main/L0_MergeRequest_PR pipeline #67788 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

xwang233 added 12 commits October 31, 2026 21:16

xwang233 requested review from a team, Wanli-Jiang and WeiHaocheng as code owners October 31, 2026 21:26

github-actions Bot assigned xwang233 Oct 31, 2026

xwang233 requested review from a team and litaotju as code owners October 31, 2026 21:27

xwang233 requested review from Mgluhovskoi, QiJune, allisonlim-nv, bo-nv, chienchunhung, lowsfer and thorjohnsen and removed request for a team October 31, 2026 21:27

xwang233 assigned Tabrizian Oct 31, 2026

Tabrizian requested a review from a team as a code owner November 2, 2026 00:20

Tabrizian requested review from EmmaQiaoCh and mlefeb01 November 2, 2026 00:20

Tabrizian added 2 commits November 2, 2026 18:49

Tabrizian added the api-compatible Accepted LLM API contract change that is backwards-compatible label Nov 2, 2026

JunyiXu-nv mentioned this pull request Nov 3, 2026

[TRTLLM-17262][refactor] Move the disaggregated service types and latency logger out of responses_utils #21510

Draft

1 task done

zheyuf mentioned this pull request Nov 3, 2026

[None][feat] DFlash: TRTLLM drafter attention by default wherever it runs, cold-page compression with a TRTLLM drafter, and acceptance-length tests #21535

Draft

1 task

Conversation

xwang233 commented Oct 31, 2026

Description

Test Coverage

PR Checklist

Uh oh!

Tabrizian commented Nov 1, 2026

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

Tabrizian commented Nov 1, 2026

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

Tabrizian commented Nov 2, 2026

Uh oh!

tensorrt-cicd commented Nov 2, 2026

Uh oh!

tensorrt-cicd commented Nov 2, 2026

Uh oh!

Tabrizian commented Nov 2, 2026

Uh oh!

tensorrt-cicd commented Nov 3, 2026

Uh oh!

tensorrt-cicd commented Nov 3, 2026

Uh oh!

Tabrizian commented Nov 3, 2026

Uh oh!

tensorrt-cicd commented Nov 3, 2026

Uh oh!

tensorrt-cicd commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants