Skip to content

crc: run crc32 alongside the folding in crc32_iscsi on Zen 5 - #449

Closed
ptweezy wants to merge 1 commit into
intel:masterfrom
ptweezy:x86-crc32-iscsi-zen5
Closed

ptweezy wants to merge 1 commit into
intel:masterfrom
ptweezy:x86-crc32-iscsi-zen5

Conversation

@ptweezy

@ptweezy ptweezy commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

On Zen 5, crc32_iscsi_by16_10 is limited by vpclmulqdq (one zmm instruction every two cycles), while crc32, which Zen 5 runs twice per cycle, sits idle. This PR adds crc32_iscsi_by16_10_fused, the same source built with FUSED_CRC32. For buffers of at least 4 KB, it folds 1792 bytes of each 3584 byte chunk and runs crc32 over the rest in eight 224 byte streams.

Which CPUs run it

crc32_iscsi() selects it where it would select crc32_iscsi_by16_10 and the CPU is AMD family 1Ah (base family Fh, extended family 0Bh), models 00h-2Fh, 40h-4Fh, 60h-7Fh or D0h-D7h, the Zen 5 models in Linux's arch/x86/kernel/cpu/amd.c. The model check is needed because Zen 6 is also family 1Ah in amd.c; it stays on crc32_iscsi_by16_10 until someone measures it. Zen 4, Intel and Hygon CPUs keep crc32_iscsi_by16_10 too. isa-l already selects code by CPU model: mbin_dispatch_init8_hygon sends ec_encode_data_update to the AVX version on Hygon 1/2/3, and the aarch64 crc32_iscsi and crc32_gzip_refl dispatchers check the core ID for Neoverse N1, Cortex-A57 and Cortex-A72. mbin_dispatch_init_clmul_fused runs mbin_dispatch_init_clmul and overrides its choice only on a matching CPU, so the feature checks stay in one place.

Output does not change. Below 4 KB the new function runs the same code paths as crc32_iscsi_by16_10, which assembles to the same code as on master.

Results

GB/s, crc32_iscsi_by16_10 (master) → crc32_iscsi_by16_10_fused. Ryzen 7 9800X3D: crc32_iscsi() in master and PR builds, median of three runs. Runners: both functions called directly in one process, alternating, best of 21 rounds; median over every Linux job on that CPU in the first, kernel and Intel runs, three runs per job. Only the Zen 5 rows run the new function with this PR.

CPU 4 KB 64 KB 1 MB 4 MB
Ryzen 7 9800X3D (Zen 5) 83 → 130 82 → 142 82 → 114 80 → 92
EPYC 9V45 (Zen 5, model 02h), 6 runs 68.8 → 108.8 68.3 → 119.1 68.7 → 107.2 67.8 → 85.5
EPYC 9V74 (Zen 4), 9 runs 57.4 → 51.8 58.6 → 51.4 58.3 → 48.4 58.2 → 46.9
Xeon 8370C (Ice Lake), 3 runs 50.2 → 54.3 53.2 → 53.6 53.5 → 53.3 33.6 → 32.7
Xeon 8573C (Emerald Rapids), 9 runs 94.9 → 54.4 98.2 → 49.7 98.4 → 49.0 30.9 → 31.4
Xeon 6973P-C (Granite Rapids), 3 runs 128.4 → 74.2 128.9 → 67.2 132.8 → 66.4 30.2 → 30.1

Zen 4, Emerald Rapids and Granite Rapids are 10% to 50% slower with it up to 1 MB.

Testing

  • Linux x86-64: make -f Makefile.unx check, tools/check_format.sh and test_nmake_file pass (no MSVC build).
  • Native Windows 11 on the 9800X3D: mingw make check passes, crc32_iscsi() selects the new function, and xmm6-xmm15 and the callee-saved GPRs survive the first, dispatching call and every call to the new function.
  • A checker compares crc32_iscsi(), crc32_iscsi_by16_10() and crc32_iscsi_by16_10_fused() with crc32_iscsi_base() for all lengths up to 30,000 bytes, around chunk boundaries and for random lengths up to 16 MB, next to PROT_NONE pages. It passes on the 9800X3D and the five runner CPUs.
  • The real crc_multibinary.asm, run against faked CPUID for every family/model encoding and 11 vendor strings, selects the new function for exactly the 104 Zen 5 encodings and never for Zen 6, Zen 4, Intel or Hygon.

On AMD Zen 5, crc32_iscsi_by16_10 is limited by vpclmulqdq, which
issues one zmm instruction every two cycles there, while the crc32
instruction, which Zen 5 runs twice per cycle, sits idle.

Add crc32_iscsi_by16_10_fused, built from the same source with
FUSED_CRC32. For buffers of at least 4 KB, it splits each 3584 byte
chunk into 1792 bytes for the folding loop and eight 224 byte streams
for crc32, 32 crc32 per fold iteration. The stream crcs are shifted
into place with two vpclmulqdq and xored into the second block after
the streams like an initial crc, so the folding of one block hides
their latency.

Dispatch to it where crc32_iscsi_by16_10 would be used on AMD family
1Ah (base family Fh, extended family 0Bh), models 00h-2Fh, 40h-4Fh,
60h-7Fh and D0h-D7h, the Zen 5 models in Linux's amd.c. Zen 6 is also
family 1Ah and keeps crc32_iscsi_by16_10 until someone measures it.
Other CPUs keep it too: one crc32 per cycle, as on current Intel CPUs,
cannot keep up with 32 per iteration.

On a Ryzen 7 9800X3D, crc32_iscsi goes from 83 to 130 GB/s for 4 KB
buffers, from about 82 to 137-148 GB/s for 16 KB to 256 KB, from 82
to 114 GB/s for 1 MB, and from 80 to 92 GB/s for 4 MB. Shorter buffers
do not change.

Signed-off-by: Parker Loflin <parker@cronstable.dev>
@ptweezy
ptweezy force-pushed the x86-crc32-iscsi-zen5 branch from c1b6b3f to b8516a6 Compare October 21, 2026 21:27
@ptweezy
ptweezy marked this pull request as draft October 21, 2026 22:01
@ptweezy ptweezy closed this Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant