Skip to content

Bundle direct2560 grouped matmul for Ascend - #351

Draft
xieminghui187 wants to merge 1 commit into
DeepLink-org:mainfrom
xieminghui187:agent/grouped-matmul-direct2560-pybind
Draft

Bundle direct2560 grouped matmul for Ascend#351
xieminghui187 wants to merge 1 commit into
DeepLink-org:mainfrom
xieminghui187:agent/grouped-matmul-direct2560-pybind

Conversation

@xieminghui187

Copy link
Copy Markdown

Summary

  • bundle the Ascend grouped-matmul direct2560 operator copied from the DLBlas implementation
  • expose the ACLNN operator through pybind on the current NPU stream
  • register the bundled, renamed custom OPP before importing torch_npu to avoid global operator collisions
  • add a contiguous 2560-expert execution path that reduces each MoE MLP from six chunked grouped matmuls to two grouped matmuls
  • retain the existing chunked implementation as the default; enable this path with DLINFER_GMM_EXPERIMENT=direct2560

The branch is based directly on the requested DLInfer main commit 92cd7982ccb142a18278be6692a6142740099631.

Implementation notes

  • supports FP16/BF16, 2560 logical experts, and type-1 tensor group lists
  • uses a renamed DlinferGroupedMatmulDirect ACLNN/op-api entry and bundled Ascend 910_93 artifacts
  • fixes repeated-call teardown by respecting aclTensorList ownership and releasing the temporary unique_ptr guards
  • includes a direct probe and unit coverage

Validation

  • FP16 correctness: max_abs=0.0
  • BF16 correctness: max_abs=0.0
  • BF16 repeated-call test: 100 iterations passed
  • TP=2 LMDeploy smoke test passed
  • ShareGPT benchmark: 500/500 requests succeeded
  • output throughput: 1025.56 tok/s -> 1161.19 tok/s (+13.22%)
  • Black, clang-format, Python compilation, and git diff --check passed

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants