Proposal (Unofficial, not merge-ready): DeepSeek-V4-Flash 4xH200 SGLang MTP recipe / 提案（非官方，尚不可合并）：DeepSeek-V4-Flash 4×H200 SGLang MTP 配方#2329
Open
kfastino wants to merge 2 commits into
Conversation
…posal Adds a proposal-only, Unofficial single-node recipe for DeepSeek-V4-Flash on 4xH200 with SGLang: TP4, Marlin W4A16 MoE runner, and EAGLE speculative decoding (3 steps / top-k 1 / 4 draft tokens). The topology is adapted from the public SGLang Hopper cookbook recommendation, and the server flags mirror the existing dsv4_fp8_h200_sglang_mtp.sh sibling. The workload keeps stock InferenceX 8k1k semantics (8192 ISL / 1024 OSL, deterministic random tokens, ignore-EOS, 10x concurrency measured requests) and the full c1..128 concurrency sweep. The SGLang image is digest-pinned and the Hugging Face checkpoint revision is pinned so the proposal is reproducible. No benchmark numbers are claimed. A green upstream H200 full sweep with evals, NVIDIA CODEOWNER sign-off, a merged SGLang cookbook recipe link, and an authorized /reuse-sweep-run all remain outstanding. 中文：新增 DeepSeek-V4-Flash 在 4xH200 上的非官方（Unofficial）单节点 SGLang 配方提案： TP4、Marlin W4A16 MoE 运行后端，以及 EAGLE 投机解码（3 步 / top-k 1 / 4 个草稿 token）。 拓扑参考公开的 SGLang Hopper 配方，服务端参数与现有的 dsv4_fp8_h200_sglang_mtp.sh 保持一致。 工作负载沿用 InferenceX 既有的 8k1k 基准测试语义与完整的 c1..128 并发扫描；镜像按 digest 固定， 检查点 revision 也已固定，以保证可复现。本提交不声明任何基准测试数值：上游 H200 完整扫描与评估、 NVIDIA CODEOWNER 签署、已合并的 SGLang cookbook 配方链接，以及授权的 /reuse-sweep-run 均仍待完成。 Co-authored-by: Cursor <cursoragent@cursor.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
中文：将 dsv4flash H200 变更日志条目的 pr-link 指向 PR SemiAnalysisAI#2329。 Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an Unofficial, proposal-only single-node recipe for DeepSeek-V4-Flash on 4xH200 with SGLang:
--moe-runner-backend marlin(W4A16), EAGLE speculative decoding: 3 steps / top-k 1 / 4 draft tokenslmsysorg/sglang:v0.5.15.post1@sha256:289cf51da1e5fd6f8eb3231f0202d46800c2e241ecf28f68a5a51507ed928e31(upstreamlmsysorgDocker Hub, unmodified — no engine patching)deepseek-ai/DeepSeek-V4-Flash@60d8d70770c6776ff598c94bb586a859a38244f1dsv4_fp8_h200_sglang_mtp.shsiblingBenchmark semantics are stock InferenceX, unchanged: 8192 ISL / 1024 OSL, deterministic random tokens, ignore-EOS,
10 x concurrencymeasured requests, and the full c1..128 sweep (1, 2, 4, 8, 16, 32, 64, 128), with evals marked at c64/c128 by the existingmark_eval_entries()logic.Files changed
benchmarks/single_node/fixed_seq_len/dsv4flash_fp8_h200_sglang_mtp.shconfigs/nvidia-master.yamldsv4flash-fp8-h200-sglang-mtpentryperf-changelog.yamlutils/matrix_logic/test_generate_sweep_configs.pyRelated Issue
Proposal issue: #2315
Type of Change
Two decisions I would like maintainers to confirm
1. Is DeepSeek-V4-Flash in scope at all? PR #1135 was previously closed in favour of DeepSeek-V4-Pro. If Flash is still out of scope, say so and I will close this and the issue — I do not want to consume review time on an unwanted model. Everything below only matters if Flash is in scope.
2.
fp8vsfp4in the config key. I labelled thisdsv4flash-fp8-h200-sglang-mtp(precision: fp8) to match the existingdsv4-fp8-h200-sglang-mtpsibling: every H200 DeepSeek-V4 entry innvidia-master.yamlserves the stock checkpoint (which declaresquant_method: fp8) through the Marlin runner, and the neighbouring comment already notes "H200 has no FP4 path". The SGLang cookbook calls this same Hopper path "original FP4", so the two taxonomies disagree. Happy to rename todsv4flash-fp4-h200-sglang-mtpif you prefer the cookbook naming — note the key,precision, and the script filename must move together, because the H200 launchers derive the script path as<model-prefix>_<precision>_h200_<framework>_mtp.sh.Related: I set
runner: h200-dgxcrather than the broaderh200pool, becauselaunch_h200-cw.sh/launch_h200-nb.shbuild the script name without the_<framework>tag and would look for a nonexistentdsv4flash_fp8_h200_mtp.sh. Retarget as you see fit.Outstanding gates (none of these are met)
Per
CONTRIBUTING.mdanddocs/PR_REVIEW_CHECKLIST.md, this PR is not eligible for merge until:configs/nvidia-master.yaml→ @Ankur-singh / @kedarpotdar-nv / @InferenceX/core) using the current checklist template./reuse-sweep-run.perf-changelog.yamlpr-linkplaceholderI have deliberately not applied a sweep label, since I am not a maintainer and a full H200 fan-out is expensive shared GPU time. Please apply
full-sweep-fail-fastif and when you want it to run.About our local numbers — diagnostic only, not part of this PR
We ran bounded 4xH200 attempts on Modal at concurrency 1, 4, and 8. Those numbers are diagnostic and Unofficial: hand-run on non-InferenceX infrastructure, outside the authorized runner pool, and not produced by this repository's workflow. They are deliberately not committed —
results/is not a tracked directory anywhere in this repository, so committing result JSONs would be a non-workflow artifact. They are not evidence of performance and nothing in this PR depends on them; I can attach them to #2315 on request.Checklist
_zhdoc pair is affected)perf-changelog.yamlperf-changelog.yamlentries are appended to the end of the file/reuse-sweep-run— not done, and premature: there is no green sweep yetLocal validation
bash -n dsv4flash_fp8_h200_sglang_mtp.shpytest utils/matrix_logic/pytest utils/changelog_gate_tests/ utils/test_process_changelog.pyutils/validate_perf_changelog.py --base-ref maingenerate_sweep_configs.py full-sweep --model-prefix dsv4flash[1,2,4,8,16,32,64,128], evals at c64/c128git diff --checkperf-changelog.yamlis append-only in this PR (7 lines added, 0 removed, no whitespace touched).中文说明
变更内容
新增 DeepSeek-V4-Flash 在 4xH200 上的**非官方（Unofficial）**单节点 SGLang 配方（仅为提案）：
--moe-runner-backend marlin（W4A16）、EAGLE 投机解码：3 步 / top-k 1 / 4 个草稿 tokenlmsysorg/sglang:v0.5.15.post1@sha256:289cf51d...（来自上游lmsysorg官方仓库，未做任何修改，不存在对推理引擎打补丁的情况）deepseek-ai/DeepSeek-V4-Flash@60d8d707...dsv4_fp8_h200_sglang_mtp.sh保持一致基准测试语义完全沿用 InferenceX 现有实现，未做改动：8192 ISL / 1024 OSL、随机 token、ignore-EOS、
10 × 并发数的测量请求，以及完整的 c1..128 并发扫描（1, 2, 4, 8, 16, 32, 64, 128），评估由现有的mark_eval_entries()逻辑标记在 c64/c128。希望维护者确认的两个问题
1. DeepSeek-V4-Flash 是否在收录范围内？ 此前 PR #1135 已被关闭，转而采用 DeepSeek-V4-Pro。如果 Flash 仍不在范围内，请直接告知，我会关闭本 PR 和对应 issue，不占用额外的评审资源。
2. 配置键使用
fp8还是fp4？ 我采用dsv4flash-fp8-h200-sglang-mtp（precision: fp8），以与现有的dsv4-fp8-h200-sglang-mtp保持一致：nvidia-master.yaml中所有 H200 的 DeepSeek-V4 配置都通过 Marlin 运行后端加载原始检查点（其quant_method声明为fp8），且相邻注释已说明 "H200 has no FP4 path"。而 SGLang cookbook 将同一条 Hopper 路径称为 "original FP4"，两套命名口径不一致。如果维护者更倾向 cookbook 的命名，我可以改名为dsv4flash-fp4-h200-sglang-mtp—— 需要注意配置键、precision与脚本文件名必须同步修改，因为 H200 启动器按<model-prefix>_<precision>_h200_<framework>_mtp.sh推导脚本路径。此外，我将
runner设为h200-dgxc而非更宽泛的h200资源池：launch_h200-cw.sh/launch_h200-nb.sh在拼接脚本名时不带_<framework>后缀，会去寻找并不存在的dsv4flash_fp8_h200_mtp.sh。维护者可按需调整。仍未满足的门槛（以下均未完成）
依据
CONTRIBUTING.md与docs/PR_REVIEW_CHECKLIST.md，本 PR 在满足以下条件前不具备合并资格：configs/nvidia-master.yaml→ @Ankur-singh / @kedarpotdar-nv / @InferenceX/core），并使用当前版本的检查清单模板。/reuse-sweep-run。perf-changelog.yaml的pr-link占位符我特意没有添加任何 sweep 标签：我并非维护者，而完整的 H200 扫描会占用大量共享 GPU 资源。如需运行，请由维护者添加
full-sweep-fail-fast。关于我们本地的测试数据——仅供诊断，不属于本 PR
我们在 Modal 上以并发 1、4、8 进行了有界的 4xH200 尝试。这些数据仅供诊断，且为非官方（Unofficial）：由人工在非 InferenceX 基础设施上运行，不在授权运行器资源池内，也并非由本仓库的工作流产生。这些结果文件已刻意不提交——
results/在本仓库中并非受版本控制的目录，提交结果 JSON 属于非工作流产物。它们不构成任何性能证据，本 PR 也不依赖它们；如有需要，我可以将其附在 #2315 中。本地验证
bash -n通过；pytest utils/matrix_logic/226 项通过（224 项原有 + 2 项新增）；pytest utils/changelog_gate_tests/等 89 项通过；validate_perf_changelog.py通过；配置生成得到 8 条目、并发[1,2,4,8,16,32,64,128]、评估位于 c64/c128；git diff --check无异常；针对 diff 的密钥/令牌/临时 URL 扫描无异常。perf-changelog.yaml在本 PR 中为纯追加（新增 7 行，删除 0 行，未改动任何空白字符）。