[SM120] Sync DS4 with upstream main + validated perf backports

[alexbi29]

Summary

Merge vllm-project/vllm:main at 537af2c3a4ba7462ddc9bc94ec7a4ea496da6d2e into jasl/vllm:codex/ds4-sm120-min-enable at 9ad62027bc84ca0ccbcc40853179312de770220c.

This is a draft integration PR, not a production promotion. The merge commit is 25900c381414d7d8bad3ca8e0f5d1782134a8b85, with both original histories retained. The merge required resolving 46 conflicted files, plus semantic overlaps in automatically merged files.

Integration choices and fixes

• Retain the SM120 sparse-attention paths, reusable padded-Q / packed C128A buffers, DSpark optimizations, and compact CPU offload; incorporate upstream Vision visibility and current workspace/JIT APIs.
• Keep DeepSeek V4's non-breakable graph default on SM12x while retaining upstream defaults elsewhere. Remove the obsolete eager scratch-pool implementation.
• Preserve token-ID and expert-map bounds checks alongside the Vision routing sentinels. Add a regression covering invalid IDs and valid Vision/special-token routing in both specialized and generic hash kernels.
• Preserve explicit unscaled draft RoPE with both flat and nested checkpoint RoPE parameters.
• Use the upstream draft-vocabulary sampling hook before Markov bias/scattering; add a reduced-vocabulary regression. Adapt the confidence-head and weight-finalization hooks to the fork's flat DeepSeek draft class.
• Keep the direct-top-k memory-saving path out of distributed decode paths that require logits for global candidate merging.
• Port compact accounting to upstream's KVCacheTensor.layers / layer_stride representation, including aliased cache groups. Use the resolved layout to distinguish packed from layer-major allocations. Add a per-layer-stride accounting regression.
• Retain upstream CPU-offload access accounting at store-offer time, new cache-manager types, protocol-module moves, and FlashInfer 0.6.18.

Duplicate-work check

Checked open PRs on jasl/vllm immediately before publishing. #44 is a focused SM12x MQA dispatch fix and #46 concerns CPU mmap registration; neither is an upstream-main integration PR. This PR does not cherry-pick either open PR. It is intended for this fork branch, not as another replacement for vllm-project/vllm#41834.

Validation

Performed in a new worktree and independent virtual environment, not the production installation, on an RTX PRO 6000 Blackwell / SM120 host.

• Full clean native editable build passed: CUDA 13.3, SM120 target, PyTorch 2.13.0+cu130, FlashInfer 0.6.18. C++/CUDA extensions and the Rust extension built successfully.
• uv pip check --python .venv/bin/python: all installed packages compatible.
• Ruff 0.14.0 check and format check passed for the Python delta against upstream; git diff --check passed.
• .venv/bin/python tools/generate_versions_json.py --check: Docker dependency metadata in sync.
• Focused unit regression suite: 309 passed, 25 deselected. Covers DSpark/config/rope, ghost and single-type KV managers, compact accounting/allocators/policies, tokenizer/thinking kwargs, and model metadata. GPU-dependent cases were explicitly deselected after production reloaded; they are not counted as passing in this run.
• Connector/config and compact transport: 75 passed.
• GPU CPU-offload transfer tests (test_swap_blocks_batch.py, test_compact_transfer.py, test_compact_worker_spec.py, test_gpu_worker.py): 92 passed, 1 skipped.
• Hash routing / Vision-bias tests: 6 passed, 1666 deselected.
• Initial sparse-attention/kernel suite: 26 passed, 34 skipped, 7 failed during gated Llama fixture access. A temporary synthetic local Llama config was subsequently used for metadata-only checks; 3 passed, including the updated SWA spec and short-prefill split tests. This is not a Llama model evaluation.
• Compact-accounting / MoE metadata retest after fixture and layout fixes: 25 passed.

Representative commands

CUDA_HOME=/usr/local/cuda-13 TORCH_CUDA_ARCH_LIST=12.0 \
  CMAKE_ARGS=-DCMAKE_CUDA_ARCHITECTURES=120 MAX_JOBS=48 NVCC_THREADS=1 \
  TMPDIR=/dev/shm uv pip install --python .venv/bin/python -e . \
  --no-build-isolation --no-deps -v

.venv/bin/python -m pytest -q --tb=short \
  tests/v1/kv_connector/unit/offloading_connector/test_config.py \
  tests/v1/kv_offload/test_compact_transport.py

.venv/bin/python -m pytest -q --tb=short \
  tests/v1/kv_offload/cpu/test_swap_blocks_batch.py \
  tests/v1/kv_offload/cpu/test_compact_transfer.py \
  tests/v1/kv_offload/cpu/test_compact_worker_spec.py \
  tests/v1/kv_offload/cpu/test_gpu_worker.py

.venv/bin/python -m pytest -q tests/kernels/moe/test_topk_softplus_sqrt.py \
  -k 'hash_bounds_guard or hash_bias_vl'

309-pass focused unit command

.venv/bin/python -m pytest -q --maxfail=15 \
  tests/config/test_deepseek_v4_cudagraph_config.py \
  tests/models/test_deepseek_v4_rope.py \
  tests/models/test_deepseek_v4_mega_moe.py \
  tests/models/test_deepseek_v4_fi_moe_ep.py \
  tests/models/test_deepseek_v4_nvfp4_draft_routing.py \
  tests/models/test_dspark_v2_speculator_hooks.py \
  tests/models/test_dspark_shared_expert_pad.py \
  tests/v1/spec_decode/test_dspark_config.py \
  tests/v1/spec_decode/test_dspark_aux_layer_ids.py \
  tests/transformers_utils/test_dspark_mla_config.py \
  tests/transformers_utils/test_speculators_dspark_config.py \
  tests/v1/core/test_ghost_block_guard.py \
  tests/v1/core/test_single_type_kv_cache_manager.py \
  tests/v1/kv_offload/cpu/test_compact_manager.py \
  tests/v1/kv_offload/cpu/test_compact_accounting.py \
  tests/v1/kv_offload/cpu/test_fixed_page_allocator.py \
  tests/v1/kv_offload/cpu/test_canonical_layout.py \
  tests/v1/kv_offload/cpu/test_manager.py \
  tests/v1/kv_offload/cpu/policies \
  tests/entrypoints/openai/test_deepseek_v4_thinking_kwargs.py \
  tests/tokenizers_/test_deepseek_v4.py \
  tests/model_executor/test_deepseek_v4_sparse_mla_metadata.py \
  tests/model_executor/test_deepseek_v4_kernel_warmup.py \
  tests/model_executor/test_deepseek_v4_flashmla_decode_dispatch.py \
  tests/model_executor/test_deepseek_v4_moe_metadata.py \
  --deselect=tests/models/test_deepseek_v4_mega_moe.py \
  -k 'not gpu_roundtrip and not cross_topology_roundtrip and not writer_rotation_submits and not mhc_warmup_drives_dummy_runs'

Remaining gate

• Human semantic-resolution and maintainer review remain required before merging. End-to-end 0731 serving validation and deployment are complete as recorded in the September 12 follow-up below; Vision-specific validation remains historical and separate.

AI assistance

OpenAI Codex assisted with conflict resolution, semantic review, regression tests, the isolated build, and this draft. This PR is being opened at the fork owner's explicit request; it is not represented as having completed human line-by-line review.

September 12 follow-up: selected perf backports promoted

This section supersedes the older Remaining gates deployment statements above.

The head is now abda028. Three Python-only commits were added after the validated upstream-sync head 3361f28:

• a3a2f5f: semantically adapted upstream [Spec Decode][Perf] Optimize DSpark Markov head with addmm vllm-project/vllm#50737 dense DSpark Markov projection/add fusion.
• df66c70: semantically adapted upstream [Core][Perf] Zero new KV blocks by cache group vllm-project/vllm#52187 KV zeroing by cache group.
• abda028: deepseek_v4: route unclean MQA logits through the SM12x Triton kernel #44 SM12x Triton dispatch for unclean long-prefill MQA logits.

Combined validation: 41/41 DSpark/SM120 tests; 177/178 broad KV/cache tests, with the only failure caused by gated google/gemma-3-1b-it fixture access returning HTTP 403; Ruff/format clean for all 17 changed Python files; cold boot; exact 187; structured tool call; health 200; unchanged 9.63 GiB / 1,778,916-token GPU KV allocation; native 32 GiB CPU offload; matched hard8 x15 exactly 115/120 with the same five length caps and no incorrect completed answers. Reverse-A/B warm decode was +1.04% mean. Repeated short prefill was effectively flat. A 141,317-token extreme-disambiguation recall prompt completed correctly in 26.9 seconds without truncation.

Upstream vllm-project#54674 was tested but intentionally excluded: it reduced KV capacity by 3.8%, slowed C=1 decode about 3%, and did not show a repeatable C=8 benefit.

The build is live on epyc from /home/ubuntu/llm-src-ds4-upstream-20260912. Rollback branch backup/ds4-sm120-upstream-20260912-pre-perf retains 3361f28.

[alexbi29]

2026-09-12 upstream refresh

• Fast-forwarded the existing PR head to 3b31cc0f6d; merge commit 6a40b7ecef brings upstream main through 7ee8a6dd01 (266 upstream commits).
• Resolved 34 textual conflicts across 39 remerge-resolution paths, retaining the SM120/DS4 integration while incorporating current upstream DSv4.1, KV-cache/offload, warmup, graph, FlashInfer, and speculative-decoding work.
• Added three follow-up fixes for compact offload signature validation, V4.1 mHC warmup gating, and hybrid KV-cache/coordinator interface plus tail-block accounting.
• Fresh CUDA 13.4 / SM120 native build passed and installed as vllm==0.28.1rc1.dev1066+g3b31cc0f6.cu134; dependency check, seven native extension imports, and all four fused DSv4 op schemas passed.
• Local tests: prefix caching 156 passed; compact/offload 125 passed (1 CUDA-memory-only deselection); adjacent single-type/prefix 81 passed; DSpark 37 passed; compact transport 25 passed; adaptive verification 13 passed; cudagraph manager 11 passed (1 CUDA-memory-only deselection). KV-cache utils were 110 passed with one host-topology failure because the test requests PP=4 on this 2-GPU host.
• The live DeepSeek service was not interrupted. GPU numerical kernel and live-serving validation remain intentionally pending because both GPUs are occupied by production.

[alexbi29]

2026-09-12 live validation and promotion update

The refreshed branch is now at 3361f28a78 and live on epyc after an idle, rollback-safe transition. The first cold load found two semantic sync defects that static tests missed:

• 859910d6cb forwards upstream's required num_hash_layers boundary into DeepseekV4MoE; constructor regression added.
• 3361f28a78 derives paged-MQA warmup width from configured DCP after upstream removed get_kv_cache_shard_count; regression added.

Validation:

• fused DSv4 Q/KV CUDA suite: 193 passed
• compact transport + CUDAGraph manager + DSv4 warmup: 45 passed
• broader GPU batch: 64 passed, 4 skipped, 1 environment-only gated-Llama fixture failure (HF 403)
• two final cold boots passed; exact 187, reasoning/final routing, automatic tool call, backend health, rowwise SM120 long-context warmup, and native 32 GiB CPU KV offload verified
• KV: 9.63 GiB physical GPU / 1,778,916 derived tokens / 1.70x at 1M
• matched 3x2048 decode: 292.7 tok/s median, 298.4 average vs old-code 298.9 / 300.2 (-2.1% / -0.6%)
• 1K/4K/16K/64K prefill: 6,280 / 8,272 / 7,441 / 6,563 tok/s
• hard8 x15 c2/high: 115/120; five length caps, no incorrect completed answers, request errors, or timeouts. Fresh old-code and prior merged controls were each 117/120.

Only the llama-swap stanza's source, binary, and cache paths changed; all serving flags remain matched. The previous tree/config/cache are retained for rollback. Post-push CI is running.