Skip to content

fix(models): preserve glm4 router correction bias in fp32 - #469

Open
taking-lying-flat wants to merge 1 commit into
FlashML-org:mainfrom
taking-lying-flat:fix/glm4-router-bias-fp32
Open

taking-lying-flat wants to merge 1 commit into
FlashML-org:mainfrom
taking-lying-flat:fix/glm4-router-bias-fp32

Conversation

@taking-lying-flat

Copy link
Copy Markdown
Contributor

GLM-4's resident-weight loader casts the FP32 router correction bias to BF16, and the model declaration inherits the engine's default dtype. Upcasting the bias in _route() cannot recover the lost precision and can change which experts are selected near a top-k boundary.

Declare e_score_correction_bias as FP32 and load it as FP32 so engine materialization preserves it. Correct the comments that describe rounding as safe for expert selection. This change is limited to the GLM-4 path.

Validation: CPU only, Intel Core i7-12800HX, Linux x86_64, PyTorch 2.11.0+cu130; synthetic FP32 bias with BF16 gate/input, no model checkpoint or GPU execution.

  • Local loader suite: 11 passed, including two synthetic regression cases kept outside this PR. Both cases failed before the fix: top-1 selected [0] instead of [1], and top-2 selected [2, 0] instead of [2, 1]. Comparisons are strict, with no dependence on tie-breaking. The checks also cover the real model declaration, loader, engine materialization, and unbiased routing weights.
  • Exact local command: PYTHONPATH=/home/cherry-cloud/FreeToken-glm-router-bias/python /tmp/freetoken-glm-shared-venv/bin/python -m pytest /tmp/freetoken-glm-bias-audit/test_models_loader.py -q --tb=short.
  • git diff --check passed. Real-checkpoint disagreement rates and end-to-end model quality have not been measured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant