Skip to content

Add gated GLM-5.2 bit-exact compatibility paths - #4

Draft
zrr1999 wants to merge 9 commits into
PFCCLab:mainfrom
zrr1999:glm52-bit-exact-alignment
Draft

Add gated GLM-5.2 bit-exact compatibility paths#4
zrr1999 wants to merge 9 commits into
PFCCLab:mainfrom
zrr1999:glm52-bit-exact-alignment

Conversation

@zrr1999

@zrr1999 zrr1999 commented Jul 27, 2026

Copy link
Copy Markdown
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Draft。只改 Megatron-LM。三个兼容 flag 和 native_unfused_adamw 默认 False,关闭时走原来的 fused/native 路径。

  • --norm-accuracy-compatible:显式 fp32 RMSNorm。DSA qk-norm 与 MTP layer-norm spec 可选用。
  • --dsa-accuracy-compatible:关掉 fused DSA kernel;unfused 路径用显式 masked softmax backward。
  • --router-accuracy-compatible:router gating 用 fp32 torch.mm
  • native_unfused_adamwtorch.optim.AdamW(foreach=False, fused=False)

MTP spec 把 config 传入 get_mtp_layer_spec*,否则 MTP 选不到上面的 RMSNorm。

测试:RMSNorm 兼容路径、DSA unfused softmax backward、router fp32 gating、MTP spec 使用传入的 config。

Issue tracking

Linked issue: 无

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant