[Klaud Cold] Add GLM-5.2 NVFP4 GB300 agentic disaggregated dynamo-sglang recipe / 新增 GLM-5.2 NVFP4 GB300 智能体分离式 dynamo-sglang 配置#2299
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：将 perf-changelog 的 pr-link 更新为 #2299。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29903146205 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29904819795 |
|
✅
中文： |
e7e0a78 to
d6a23b0
Compare
Oseltamivir
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29911592386 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29912225195 |
d6a23b0 to
c3a1b08
Compare
Oseltamivir
changed the title
c3a1b08 to
2bf8101
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29913501344 |
2bf8101 to
fd36093
Compare
Oseltamivir
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29920754828 |
fd36093 to
225c439
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29921699960 |
225c439 to
1f7c557
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29923390686 |
1f7c557 to
a10a67b
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29924405261 |
Agentic-coding disaggregated (prefill/decode) GB300 sweep, modeled on dsv4-fp4-gb300-dynamo-sglang-agentic-disagg (run 29385297092): EAGLE/MTP + DRAM hierarchical-cache KV offload, mooncake KV transfer, dedicated etcd/nats node + session-affinity dynamo frontend, custom agentic_srt.sh benchmark. Wide-EP decode (1D TP16/EP16 attention-DP) fed by TP8/EP8 attention-DP prefill workers scaling 1->6; arms conc 8 (D TP4), 256, 256, 512, 1024 (3/6/8/10/16 nodes). GLM-5.2 serve flags (glm47/glm45 parsers, deepep, hicache-ratio 0.75, SGLANG_TIMEOUT_KEEP_ALIVE=900 for the AIPerf warmup race) grafted onto the DSv4 agentic disagg machinery. Image v0.5.15.post1-cu130, dynamo 1.3.0.dev1 (>=1.2.1 needed for GLM-5.2's chat-template `.0` dot-index). - Recipes benchmarks/multi_node/srt-slurm-recipes/sglang/glm5.2/gb300-fp4/agentic/*.yaml - configs/nvidia-master.yaml: glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg (runner cluster:gb300-nv, mooncake, dynamo-router 1.3.0.dev1) - configs/runners.yaml: add cluster:gb300-nv hardware entry (available-cpu-dram-mib/gpus-per-node) required by agentic DRAM-offload sizing; also unblocks the existing DSv4 gb300 agentic configs - launch_gb300-nv.sh: IS_AGENTIC glm5.2 dynamo-sglang branch (srt-slurm v1.0.10) + glm5.2 model-path (/data shared-NFS shim) - perf-changelog.yaml trigger entry 中文：GB300 智能体(agentic)分离式(预填充/解码)dynamo-sglang 配方，参照 dsv4-fp4-gb300-dynamo-sglang-agentic-disagg（run 29385297092）：EAGLE/MTP + DRAM 分层缓存 KV 卸载、mooncake KV 传输、专用 etcd/nats 节点 + 会话亲和 dynamo 前端、自定义 agentic_srt.sh 基准。宽 EP 解码（1D TP16/EP16 注意力 DP），由 TP8/EP8 注意力 DP 预填充 worker（1→6）驱动。 GLM-5.2 服务标志（glm47/glm45 解析器、deepep、hicache-ratio 0.75）叠加于 DSv4 智能体分离式 机制之上。新增 cluster:gb300-nv 硬件条目（智能体 DRAM 卸载所需，同时修复现有 DSv4 配置）。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
a10a67b to
14593e3
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29925753943 |
Resolve perf-changelog.yaml by taking main's bytes verbatim and re-appending this PR's glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg entry at the tail (KLAUD_DEBUG.md 1.1) -- a 3-way merge silently rewrites whitespace and re-trips the changelog deletion check. 中文：将 origin/main 合并入本分支。perf-changelog.yaml 按 KLAUD_DEBUG.md 1.1 处理：直接取 main 的原始字节，再把本 PR 的 glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg 条目追加到文件末尾——三方合并 会静默改动空白字符并再次触发删除检查。
…v0.5.16
Every prefill worker in the GB300 agentic disagg sweep died on its first
forward (run 29925753943):
File "sglang/srt/layers/moe/moe_runner/flashinfer_cutedsl.py", line 476,
in fused_experts_deepep_to_flashinfer_cutedsl_fp4
hidden_states, hidden_states_scale, _, _, masked_m, _ = dispatch_output
ValueError: not enough values to unpack (expected 6, got 5)
The only ("deepep", *) NVFP4 MoE fused func SGLang registers is
flashinfer_cutedsl's, and it consumes the 6-field low-latency
DeepEPLLDispatchOutput. `deepep-mode: auto` resolves to `normal` on every
extend batch (DeepEPMode.resolve), which yields the 5-field
DeepEPNormalDispatchOutput -- so a prefill-role engine can never survive
deepep + NVFP4. `deepep-mode: low_latency` is not a way out either: DeepEP
asserts num_max_dispatch_tokens_per_rank <= 1024 while chunked prefill puts
32768/dp8 = 4096 tokens on a rank. This is unchanged in v0.5.16, so it is a
recipe bug, not an image bug.
Both roles move to the MoE backends the proven in-repo NVFP4 recipes use:
prefill no EP a2a + moe-runner-backend flashinfer_trtllm
(qwen3.5/gb300-fp4 maxtpt, dsr1/b200-fp4 1k1k disagg mtp)
wide-EP moe-a2a-backend flashinfer (MNNVL) + flashinfer_cutedsl
decode (qwen3.5/gb300-fp4 maxtpt). The ("flashinfer", cutedsl) fused
func takes a StandardDispatchOutput, so unlike the deepep one
it also survives MTP's extend-shaped target-verify batch.
ep-num-redundant-experts / ep-dispatch-algorithm are dropped along with
DeepEP: the FlashInfer a2a dispatcher takes num_local_experts as "Unused" and
plumbs no expert map, so they had no effect on this path. Worth revisiting as
a tuning follow-up. SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK is replaced
by SGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK on the decode role,
which sizes the a2a recv buffer (peak 3072/16 * 4 = 768 tokens/rank).
Image bumped v0.5.15.post1-cu130 -> v0.5.16-cu130 in the same pass.
A second, separate failure surfaced once prefill was fixed: the decode engine
died at init with "Pre-permute function for flashinfer to triton is not
registered". GLM-5.2-NVFP4 leaves the MTP/nextn layer unquantized
(hf_quant_config excludes `model.layers.78*`), so the EAGLE draft's MoE is bf16
and UnquantizedFusedMoEMethod pins it to the triton runner core; inheriting the
target model's FlashInfer a2a then has no ("flashinfer","triton") permute.
SGLang already handles this in _deepseek_spec_moe_resolution ("use triton fused
moe by default for bf16 nextn layer in deepseek fp4 checkpoint") but gates the
hook on is_hip(), so on CUDA the draft silently inherits. The wide-EP decode
now sets upstream's own ROCm values explicitly:
speculative-moe-a2a-backend none + speculative-moe-runner-backend triton.
Validated in real disaggregation on GB300 batch_1 before CI, via a replica of
runners/launch_gb300-nv.sh (srt-slurm v1.0.10, srtctl apply, mooncake KV
transfer, dedicated etcd/nats node, session-affinity dynamo frontend) on the
2p4d c256 arm -- the smallest arm exercising both changed roles:
job 17402 deepep prefill -> died, but on the NEW decode-side
Pre-permute assert; the original
unpack ValueError was already gone
job 17430 this recipe, 2P x 4D -> "Server is healthy - starting
benchmark" (1P + 1D up in 9m10s)
job 17431 trtllm-decode fallback -> also healthy; kept as the
documented alternative
Both jobs then failed only in the benchmark step, because the out-of-CI replica
does not export KV_OFFLOAD_BACKEND (CI sets it from the master config's
kv-offload-backend). Engine bring-up is what was under test and it passed. The
c8 arm's TP4/EP1 decode was not run on-node; it is unaffected by both changes
(a2a none on both roles) and its prefill is identical to the arms that were.
中文：GB300 智能体分离式扫描中每个预填充(prefill) worker 在首次前向传播即崩溃
（run 29925753943）。SGLang 注册的唯一 ("deepep", *) NVFP4 混合专家(MoE)融合
函数来自 flashinfer_cutedsl，它消费 6 字段的低延迟 DeepEPLLDispatchOutput；而
`deepep-mode: auto` 在任何 extend 批次上都解析为 `normal`，产出 5 字段的
DeepEPNormalDispatchOutput，因此预填充角色在 deepep + NVFP4 下不可能正常工作。
改用 low_latency 也走不通：DeepEP 断言 num_max_dispatch_tokens_per_rank <= 1024，
而分块预填充在单 rank 上会放 32768/dp8 = 4096 个 token。v0.5.16 中该逻辑未变，
所以这是配置问题而非镜像问题。
两个角色改用仓库内已验证的 NVFP4 配方所用的 MoE 后端：预填充不使用 EP
all-to-all，改为 moe-runner-backend flashinfer_trtllm；宽 EP 解码使用
moe-a2a-backend flashinfer（MNNVL）+ flashinfer_cutedsl——该融合函数接收
StandardDispatchOutput，因此也能承受 MTP 的 extend 形状 target-verify 批次。
ep-num-redundant-experts / ep-dispatch-algorithm 随 DeepEP 一并移除（FlashInfer
a2a 分发器不传递专家映射，这两个参数在此路径上本就无效），可作为后续调优项。
镜像同步升级至 v0.5.16-cu130。预填充修复后暴露出第二个独立问题：解码引擎在初始化阶段崩溃，报
"Pre-permute function for flashinfer to triton is not registered"。
GLM-5.2-NVFP4 未量化 MTP/nextn 层（hf_quant_config 排除了 `model.layers.78*`），
因此 EAGLE 草稿模型的 MoE 为 bf16，UnquantizedFusedMoEMethod 会强制其使用 triton
runner；此时若沿用目标模型的 FlashInfer a2a，则不存在 ("flashinfer","triton")
的 pre-permute 函数。SGLang 在 _deepseek_spec_moe_resolution 中已处理该情况
（"对 fp4 检查点中的 bf16 nextn 层默认使用 triton fused moe"），但该钩子被
is_hip() 限制，故在 CUDA 上草稿模型会静默继承目标配置。现已在宽 EP 解码中显式
设置上游在 ROCm 下使用的同一组取值：speculative-moe-a2a-backend none 与
speculative-moe-runner-backend triton。
上线 CI 前已在 GB300（batch_1）以真实分离式部署验证：复刻
runners/launch_gb300-nv.sh（srt-slurm v1.0.10、srtctl apply、mooncake KV 传输、
专用 etcd/nats 节点、会话亲和 dynamo 前端），运行 2p4d c256 这一同时覆盖两个角色
改动的最小拓扑。job 17430（本配方，2P x 4D）达到 "Server is healthy - starting
benchmark"（1P + 1D 在 9 分 10 秒内就绪）；job 17431（trtllm 解码回退方案）同样
健康，作为备选方案记录在案。两个作业随后仅在基准测试步骤失败，原因是 CI 之外的
复刻环境未导出 KV_OFFLOAD_BACKEND（CI 由主配置的 kv-offload-backend 提供）——本次
验证的目标是引擎启动，该目标已通过。c8 拓扑的 TP4/EP1 解码未在节点上实测：它不受
本次两项改动影响（两个角色均为 a2a none），且其预填充配置与已验证的拓扑完全一致。
Fixed the prefill crash from run 29925753943, rebased, image → SGLang v0.5.16Root cause: DeepEP cannot serve NVFP4 outside low-latency decodeEvery prefill worker died on its first forward: The only This is a recipe bug, not an image bug — the unpack is byte-identical in Fix: the MoE backends the proven in-repo NVFP4 recipes use
The Also dropped with DeepEP: Second root cause (surfaced only after the first was fixed)With prefill fixed, the decode engine then died at init with:
SGLang already handles exactly this — speculative-moe-a2a-backend: none
speculative-moe-runner-backend: triton
Image bump
RebaseThe PR was On-node validationVerified in real disaggregation (no aggregated stand-in) on GB300
Both then failed only in the benchmark step, because the out-of-CI replica does Not covered on-node: the 中文说明根因：DeepEP 无法在低延迟解码之外服务 NVFP4run 29925753943 中每个预填充(prefill) worker 在首次前向传播即崩溃，报 修复：改用仓库内已验证 NVFP4 配方所用的 MoE 后端
镜像升级镜像由 变基此前 PR 处于 节点验证已在 GB300 job 17430（本配方，2P TP8/EP8/DPA × 4D TP16/EP16/DPA）达到 未在节点上覆盖的部分： |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30166607252 |
1 participant
Summary
Adds a GLM-5.2 NVFP4 agentic-coding disaggregated (prefill/decode) GB300 benchmark via
dynamo-sglang, modeled ondsv4-fp4-gb300-dynamo-sglang-agentic-disagg(run 29385297092).New config-key
glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg:agentic_srt.shbenchmark1D TP16/EP16 attention-DP(deepep), fed byTP8/EP8attention-DP prefill workers scaling 1→6p1x8_d1x4conc 8,p1x8/p2x8_d1x16conc 256,p3x8_d1x16conc 512,p6x8_d1x16conc 1024 (3/6/8/10/16 nodes)Changes
benchmarks/multi_node/srt-slurm-recipes/sglang/glm5.2/gb300-fp4/agentic/*.yaml— 5 new recipes (DSv4 agentic disagg structure + GLM-5.2 serve flags:glm47/glm45parsers,deepep,hicache-ratio 0.75,SGLANG_TIMEOUT_KEEP_ALIVE=900for the AIPerf warmup race)configs/nvidia-master.yaml—glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg(runner: cluster:gb300-nv, mooncake, dynamo-router1.3.0.dev1)configs/runners.yaml— newcluster:gb300-nvhardware entry (available-cpu-dram-mib/gpus-per-node) required by the agentic DRAM-offload sizing; also unblocks the existing DSv4 gb300 agentic configs (previously errored full config-gen)runners/launch_gb300-nv.sh—IS_AGENTICglm5.2 dynamo-sglang branch (srt-slurmv1.0.10) + glm5.2 model-path (shared-NFS/datashim)perf-changelog.yaml— trigger entryImage / weights / dynamo
lmsysorg/sglang:v0.5.15.post1-cu130(GLM-5.2-validated on-node). Dynamo 1.3.0.dev1 — ≥1.2.1 is required because older dynamo's minijinja PromptFormatter rejects GLM-5.2's chat-template.0dot-index (verified against the earlier disagg run's frontend log).nvidia/GLM-5.2-NVFP4staged on shared NFS at/data/home/sa-shared/models/GLM-5.2-NVFP4.中文说明
新增 GLM-5.2 NVFP4 智能体(agentic)分离式(预填充/解码)GB300 基准测试（
dynamo-sglang），参照dsv4-fp4-gb300-dynamo-sglang-agentic-disagg（run 29385297092）。新增配置键
glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg：agentic_srt.sh基准1D TP16/EP16 注意力 DP（deepep），由TP8/EP8注意力 DP 预填充 worker（1→6）驱动改动
glm47/glm45解析器、deepep、hicache-ratio 0.75、SGLANG_TIMEOUT_KEEP_ALIVE=900）configs/nvidia-master.yaml新增glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg（runner: cluster:gb300-nv、mooncake、dynamo-router1.3.0.dev1）configs/runners.yaml新增cluster:gb300-nv硬件条目（智能体 DRAM 卸载所需；同时修复此前报错的 DSv4 gb300 智能体配置的全量配置生成）runners/launch_gb300-nv.sh新增IS_AGENTICglm5.2 dynamo-sglang 分支（srt-slurmv1.0.10）+ glm5.2 模型路径（共享 NFS/data垫层）perf-changelog.yaml触发项镜像 / 权重 / dynamo
lmsysorg/sglang:v0.5.15.post1-cu130（已在节点验证 GLM-5.2 NVFP4）。Dynamo 1.3.0.dev1——需 ≥1.2.1，否则旧版 dynamo 的 minijinja 无法解析 GLM-5.2 聊天模板的.0点索引。nvidia/GLM-5.2-NVFP4暂存于共享 NFS/data/home/sa-shared/models/GLM-5.2-NVFP4。🤖 Generated with Claude Code