Skip to content

igzip: add an AVX-512 adler32 - #445

Closed
ptweezy wants to merge 2 commits into
intel:masterfrom
ptweezy:x86-adler32-crc32-iscsi
Closed

ptweezy wants to merge 2 commits into
intel:masterfrom
ptweezy:x86-adler32-crc32-iscsi

Conversation

@ptweezy

@ptweezy ptweezy commented Oct 20, 2026 •

Copy link
Copy Markdown

This PR adds adler32_avx512, an AVX-512 VNNI adler32 with 256-bit vectors. Master's isal_adler32() uses adler32_avx2_4 on AVX2 CPUs, which adds 16 bytes per iteration into one pair of accumulators, so each iteration waits for the previous one; deflate and inflate with a zlib wrapper call it for all of their data. The new function takes 128 bytes per iteration, with one vpdpbusd accumulator per 32-byte block.

The first commit makes mbin_dispatch_init8 (and its Hygon variant), which master also uses for eight erasure code and RAID functions, keep the AVX2 function instead of the slot 8 one when CPUID reports the second AVX-512 group without the whole first group, as a VM that hides AVX512F can.

CPUs and builds

isal_adler32() picks adler32_avx512 on CPUs with both AVX-512 groups that isa-l checks for (Intel Ice Lake and later CPUs with AVX-512, Zen 4, Zen 5). Cascade Lake and AVX2-only CPUs keep adler32_avx2_4. Output does not change.

256-bit vectors

I compared a 512-bit version of the same loop by switching isal_adler32() between the versions inside deflate and inflate in one binary (harness, runs, more runs). They trade off:

  • 512-bit vectors make the function alone 1.7x to 1.9x as fast as 256-bit vectors on Zen 5 and 1.1x to 1.5x on the other runner CPUs (4 KB to 256 KB buffers), and deflate and inflate with a zlib wrapper up to 0.8 percentage points faster on Zen 4, Zen 5 and Emerald Rapids.
  • 256-bit vectors avoid the drop of about 3% in core clock that 512-bit vectors cause on Ice Lake (4 runners) and Granite Rapids (1 runner). With 512-bit vectors, deflate and inflate with a zlib wrapper run up to 5.4% slower than with adler32_avx2_4 on Granite Rapids and 1.8% slower on Ice Lake; with 256-bit vectors they run faster on every CPU tested.

This PR uses 256-bit vectors on every CPU. Using 512-bit vectors only where they help would need a dispatch by CPU model.

Results

GitHub runners, three runs per runner (run); each cell is the median over the runs of the best of 21 rounds. The kernel column times direct calls; the other columns alternate master (8fa1e72) and this PR in one process on 64 KB slices of the Silesia corpus. The runs built the adler32 commit alone; the first commit only adds a branch to code that runs once per function.

Runner CPU (runs) Kernel, 64 KB (GB/s): adler32_avx2_4 → adler32_avx512 Deflate level 1, zlib wrapper (MB/s): master → PR Inflate, zlib wrapper (MB/s): master → PR
AMD EPYC 9V45, Zen 5 (12) 14.8 → 105.8 595 → 617 (+3.7%) 831 → 870 (+4.7%)
AMD EPYC 9V74, Zen 4 (12) 18.3 → 82.0 481 → 492 (+2.3%) 772 → 777 (+0.6%)
Intel Xeon 8370C, Ice Lake (9) 18.0 → 57.2 424 → 430 (+1.5%) 646 → 653 (+1.0%)
Intel Xeon 8573C, Emerald Rapids (6) 19.9 → 58.4 428 → 436 (+1.8%) 602 → 598 (-0.5%)
Intel Xeon 6973P-C, Granite Rapids (6) 24.6 → 72.8 562 → 583 (+3.8%) 797 → 810 (+1.6%)

Without a wrapper, deflate and inflate run the same code in both libraries, yet differed by up to 3.0% at 64 KB, so the Emerald Rapids inflate change is within that noise.

Testing

  • make -f Makefile.unx check, tools/check_format.sh and test_nmake_file pass on each commit, and the mingw x86_64 make check passes natively on Windows 11 on a Ryzen 7 9800X3D. On GitHub runners, tools/test_extended.sh (EPYC 9V45) and mingw make check (four Windows runners) pass on the adler32 commit without the first commit (run). No MSVC build.
  • adler32_avx512() and isal_adler32() match adler32_base() for every length up to 20,000 bytes, random lengths up to 64 MB, buffers ending at a PROT_NONE page and a buffer over 4 GB, on the 9800X3D and the five runner CPUs. A win64 thunk test on the 9800X3D found all callee-saved registers preserved. Deflate output matches master's in every benchmark run.
  • With and without the first commit, the multibinary objects (elf64, win64) differ only by the added branch in each init8 dispatcher. Running the real dispatch code against a fake CPUID over all AVX, AVX2 and AVX-512 feature bit and XCR0 combinations changes only CPUs with the ZMM state and the second group but not the whole first group, from the slot 8 function to the AVX2 one.

@ptweezy
ptweezy force-pushed the x86-adler32-crc32-iscsi branch from d038f42 to 271faf8 Compare October 20, 2026 21:11
@ptweezy ptweezy changed the title igzip, crc: speed up x86 adler32 and crc32_iscsi igzip: add an AVX-512 adler32 Oct 20, 2026
@ptweezy
ptweezy force-pushed the x86-adler32-crc32-iscsi branch 2 times, most recently from 29dc7f3 to 2ea4cba Compare October 21, 2026 00:35
mbin_dispatch_init8 picks the AVX512 Update/10 function when CPUID
leaf 7 reports every extension in FLAGS_CPUID7_ECX_AVX512_G2 and the OS
enables the ZMM state, even when FLAGS_CPUID7_EBX_AVX512_G1 (AVX512F,
VL, BW, CD and DQ) is incomplete: the cmove of the AVX512/06 function
fails, but the cmove of the /10 function after it still runs. No real
CPU reports the second group without the first, but a VM that hides
AVX512F or AVX512VL and passes the other bits through gets a function
that needs them.

Jump to the end when the first group is incomplete, so these
configurations keep the AVX2/04 function, as in mbin_dispatch_init6.
Make the same change in mbin_dispatch_init8_hygon, which
ec_encode_data_update uses, so that it keeps expecting the table format
that ec_init_tables builds. The other users are ec_init_tables,
ec_encode_data, gf_vect_mul, gf_vect_mul_init, gf_vect_mad,
gf_vect_dot_prod and pq_gen. Only configurations with the ZMM state and
the second group but not the whole first group change.

Signed-off-by: Parker Loflin <parker@cronstable.dev>
isal_adler32() uses adler32_avx2_4 on every AVX2 CPU. It adds 16 bytes
per iteration into one pair of accumulators, so each iteration waits
for the previous one. Deflate and inflate with a zlib wrapper call it
for all of their data.

Add adler32_avx512. It takes 128 bytes per iteration in four 32 byte
blocks, with one vpdpbusd accumulator per block and vpsadbw byte sums,
reduces chunks of up to 5504 bytes modulo 65521 with 64-bit arithmetic,
and loads the last 0 to 31 bytes with a mask. It uses only ymm16 to
ymm31, so it saves no registers on Windows. isal_adler32() uses it on
CPUs with both AVX-512 groups that isa-l checks for (Ice Lake, Zen 4
and later).

The vectors are 256 bits wide: a 512-bit version lowered the core
clock on Ice Lake and Granite Rapids, and deflate or inflate with a
zlib wrapper got slower there than with adler32_avx2_4.

On GitHub runners (Zen 4, Zen 5, Ice Lake, Emerald Rapids, Granite
Rapids), it is 2.9x to 7.1x as fast as adler32_avx2_4 on 64 KB buffers.

Signed-off-by: Parker Loflin <parker@cronstable.dev>
@ptweezy
ptweezy force-pushed the x86-adler32-crc32-iscsi branch from 2ea4cba to e2da621 Compare October 21, 2026 21:27
@ptweezy
ptweezy marked this pull request as draft October 21, 2026 22:01
@ptweezy ptweezy closed this Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant