Repository navigation
Conversation
On AMD Zen 5, crc32_iscsi_by16_10 is limited by vpclmulqdq, which issues one zmm instruction every two cycles there, while the crc32 instruction, which Zen 5 runs twice per cycle, sits idle. Add crc32_iscsi_by16_10_fused, built from the same source with FUSED_CRC32. For buffers of at least 4 KB, it splits each 3584 byte chunk into 1792 bytes for the folding loop and eight 224 byte streams for crc32, 32 crc32 per fold iteration. The stream crcs are shifted into place with two vpclmulqdq and xored into the second block after the streams like an initial crc, so the folding of one block hides their latency. Dispatch to it where crc32_iscsi_by16_10 would be used on AMD family 1Ah (base family Fh, extended family 0Bh), models 00h-2Fh, 40h-4Fh, 60h-7Fh and D0h-D7h, the Zen 5 models in Linux's amd.c. Zen 6 is also family 1Ah and keeps crc32_iscsi_by16_10 until someone measures it. Other CPUs keep it too: one crc32 per cycle, as on current Intel CPUs, cannot keep up with 32 per iteration. On a Ryzen 7 9800X3D, crc32_iscsi goes from 83 to 130 GB/s for 4 KB buffers, from about 82 to 137-148 GB/s for 16 KB to 256 KB, from 82 to 114 GB/s for 1 MB, and from 80 to 92 GB/s for 4 MB. Shorter buffers do not change. Signed-off-by: Parker Loflin <parker@cronstable.dev>
ptweezy
force-pushed
the
x86-crc32-iscsi-zen5
branch
from
October 21, 2026 21:27
c1b6b3f to
b8516a6
Compare
ptweezy
marked this pull request as draft
October 21, 2026 22:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Zen 5,
crc32_iscsi_by16_10is limited byvpclmulqdq(one zmm instruction every two cycles), whilecrc32, which Zen 5 runs twice per cycle, sits idle. This PR addscrc32_iscsi_by16_10_fused, the same source built withFUSED_CRC32. For buffers of at least 4 KB, it folds 1792 bytes of each 3584 byte chunk and runscrc32over the rest in eight 224 byte streams.Which CPUs run it
crc32_iscsi()selects it where it would selectcrc32_iscsi_by16_10and the CPU is AMD family 1Ah (base family Fh, extended family 0Bh), models 00h-2Fh, 40h-4Fh, 60h-7Fh or D0h-D7h, the Zen 5 models in Linux'sarch/x86/kernel/cpu/amd.c. The model check is needed because Zen 6 is also family 1Ah inamd.c; it stays oncrc32_iscsi_by16_10until someone measures it. Zen 4, Intel and Hygon CPUs keepcrc32_iscsi_by16_10too. isa-l already selects code by CPU model:mbin_dispatch_init8_hygonsendsec_encode_data_updateto the AVX version on Hygon 1/2/3, and the aarch64crc32_iscsiandcrc32_gzip_refldispatchers check the core ID for Neoverse N1, Cortex-A57 and Cortex-A72.mbin_dispatch_init_clmul_fusedrunsmbin_dispatch_init_clmuland overrides its choice only on a matching CPU, so the feature checks stay in one place.Output does not change. Below 4 KB the new function runs the same code paths as
crc32_iscsi_by16_10, which assembles to the same code as on master.Results
GB/s,
crc32_iscsi_by16_10(master) →crc32_iscsi_by16_10_fused. Ryzen 7 9800X3D:crc32_iscsi()in master and PR builds, median of three runs. Runners: both functions called directly in one process, alternating, best of 21 rounds; median over every Linux job on that CPU in the first, kernel and Intel runs, three runs per job. Only the Zen 5 rows run the new function with this PR.Zen 4, Emerald Rapids and Granite Rapids are 10% to 50% slower with it up to 1 MB.
Testing
make -f Makefile.unx check,tools/check_format.shandtest_nmake_filepass (no MSVC build).make checkpasses,crc32_iscsi()selects the new function, and xmm6-xmm15 and the callee-saved GPRs survive the first, dispatching call and every call to the new function.crc32_iscsi(),crc32_iscsi_by16_10()andcrc32_iscsi_by16_10_fused()withcrc32_iscsi_base()for all lengths up to 30,000 bytes, around chunk boundaries and for random lengths up to 16 MB, next toPROT_NONEpages. It passes on the 9800X3D and the five runner CPUs.crc_multibinary.asm, run against faked CPUID for every family/model encoding and 11 vendor strings, selects the new function for exactly the 104 Zen 5 encodings and never for Zen 6, Zen 4, Intel or Hygon.