Skip to content

feat: support tile-misaligned GPTQ Marlin shapes - #3028

Open
ZX-ModelCloud wants to merge 1 commit into
mainfrom
zx/marlin-vllm-safe-sync
Open

feat: support tile-misaligned GPTQ Marlin shapes#3028
ZX-ModelCloud wants to merge 1 commit into
mainfrom
zx/marlin-vllm-safe-sync

Conversation

@ZX-ModelCloud

Copy link
Copy Markdown
Collaborator

Summary

  • Allow explicitly selected GPTQ Marlin layers to handle packed K/N shapes that do not already fill a 64 x 128 thread tile.
  • Zero-pad packed weights, scales, and bias during post_init, then pad activations and slice outputs around the existing Marlin GEMM.
  • Keep aligned layers on the original call path and keep runtime padding opt-in for automatic backend selection.
  • Reject partially packed dimensions and unsupported activation-order tails before kernel launch.

Compatibility

  • Supports symmetric GPTQ 4-bit and 8-bit weights with FP16 or BF16 compute.
  • Supports channelwise quantization expressed as either group_size=-1 or group_size=K.
  • Does not change CUDA sources, operator registration, or the aligned-shape execution path.
  • Tile-misaligned explicit Marlin requests may trade performance for shape compatibility; automatic selection remains conservative.

Tests

  • ruff check gptqmodel/nn_modules/qlinear/marlin.py gptqmodel/utils/marlin.py tests/test_marlin_jit.py
  • pytest -q tests/test_marlin_jit.py -m 'not cuda' (33 passed)
  • pytest -q tests/test_marlin_jit.py::test_marlin_cuda_padded_shape_matches_dequantized_reference (16 passed on sm80)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant