Skip to content

Fix QQQ workspace lifetime and reduce packing memory - #3029

Open
yujiongzhang wants to merge 4 commits into
ModelCloud:mainfrom
yujiongzhang:main
Open

Fix QQQ workspace lifetime and reduce packing memory#3029
yujiongzhang wants to merge 4 commits into
ModelCloud:mainfrom
yujiongzhang:main

Conversation

@yujiongzhang

Copy link
Copy Markdown

Summary

  • Release temporary QQQ quantization state immediately after each module is processed.
  • Replace full-size NumPy/Torch QQQ packing intermediates with bounded Torch chunked packing.
  • Preserve the existing QQQ/Marlin packed layout and quantization behavior.

Motivation

During QQQ quantization of large MoE models, CPU memory and swap usage may increase after each module or layer. The previous implementation retained QQQ quantization workspaces until later cleanup and created multiple full-size temporary tensors and NumPy arrays during weight packing.

This change releases the quantization workspace as soon as it is no longer needed and limits packing temporary memory to a fixed-size working set.

Implementation

QQQProcessor now calls QQQ.free() in a finally block after each module quantization.

QQQLinear.pack() now:

  • processes weights in bounded input/output chunks;
  • writes directly into the final CPU packed buffer;
  • uses Torch bitwise operations for INT4 packing;
  • avoids full-size NumPy packing buffers and redundant conversions;
  • preserves per-channel and grouped quantization semantics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant