Repository navigation
Conversation
ptweezy
force-pushed
the
igzip-aarch64-speedups
branch
from
October 19, 2026 00:20
8bb8958 to
806d0db
Compare
This was referenced Oct 19, 2026
ptweezy
force-pushed
the
igzip-aarch64-speedups
branch
from
October 20, 2026 21:14
bef0470 to
eb907c3
Compare
ptweezy
added a commit
to ptweezy/isa-l
that referenced
this pull request
Oct 20, 2026
Signed-off-by: Parker Loflin <parker@cronstable.dev>
ptweezy
added a commit
to ptweezy/isa-l
that referenced
this pull request
Oct 20, 2026
Signed-off-by: Parker Loflin <parker@cronstable.dev>
ptweezy
added a commit
to ptweezy/isa-l
that referenced
this pull request
Oct 20, 2026
… every level Signed-off-by: Parker Loflin <parker@cronstable.dev>
This was referenced Oct 21, 2026
ptweezy
force-pushed
the
igzip-aarch64-speedups
branch
from
October 21, 2026 21:27
eb907c3 to
259af37
Compare
ptweezy
changed the title
ptweezy
marked this pull request as draft
October 21, 2026 21:54
When a call compresses all input from the caller's buffer and the flush type is FULL_FLUSH, get_hist_size() returns 0, so isal_deflate() keeps no history. The match history is reset when sync_flush() writes the sync marker: it sets has_hist to IGZIP_NO_HIST, and the next call resets the hash table. If the output buffer fills before the end of the block is written, the flush is still pending when isal_deflate() returns, and has_hist is not IGZIP_NO_HIST. If the caller supplies more input at that point, the next call compresses that input before it writes the sync marker, with the old hash table but without the history it refers to. At every level, match candidates can then point up to 32 KB before the start of the caller's input buffer, so the compressor reads memory outside the input. When those bytes happen to match, it emits a valid deflate stream that inflates to different data. When the call that leaves the flush pending compresses from the internal buffer, which holds the history of earlier input compressed with NO_FLUSH or SYNC_FLUSH, or of a dictionary, the history stays there. If the next call's input does not fit next to it, isal_deflate() clears the flush type while it compresses, so it writes a pending sync marker without resetting the match history, and the input after the marker is compressed against the data before it. A full flush is meant to make the following blocks independent of earlier data. When a call compresses all input while a full flush is pending, set has_hist to IGZIP_NO_HIST, so that the next call resets the match history before it compresses new input, as it does after the sync marker. Stored blocks are left alone: write_stored_block() resets the match history when it completes a full flush. Callers that keep the flush type until each flush completes, and supply more input only after it completes, get the same output as before. Add two tests to igzip_rand_test. One compresses with FULL_FLUSH and small output buffers at each level, and supplies more input while a flush is pending; without the fix, it fails for most seeds, including the default one. The other leaves the sync marker of a full flush pending at the start of a stream, after a SYNC_FLUSH, and after a dictionary, then supplies IGZIP_HIST_SIZE + 8 KB more input, and checks that the data after the marker decompresses on its own. Without the fix, it fails after the SYNC_FLUSH or the dictionary for most seeds. Signed-off-by: Parker Loflin <parker@cronstable.dev>
level_buf->icf_buf_avail_out holds the free space in the ICF buffer in bytes. init_new_icf_block() and compress_icf_map_g() store bytes, every reader treats the value as bytes, and the x86 level 1 and 2 functions subtract the bytes they write. update_state() in igzip_icf_base.c and the aarch64 level 1 and 2 body and finish functions store a count of deflate_icf entries instead, so every call that leaves the block open cuts the remaining space to a quarter. The block then ends early. In stateless compression this happens when the finish function runs after the body. In stateful compression it happens each time a call runs out of input, so smaller input buffers give more blocks. On dickens at level 1, 4 KB input buffers give 45.11% instead of the 45.01% of a single call. Store the space in bytes. Stateful level 1 and 2 compression of dickens in 4 KB, 16 KB, or 64 KB input buffers then gives the same size as a single call. Signed-off-by: Parker Loflin <parker@cronstable.dev>
print_perf_line() picks the label from the deflate strategy, so the isal stateless, isal stateful, and zlib inflate runs all print the label of the deflate run before them, for example zlib_inflate for every inflate run after a zlib level. Use info->inflate_mode, which the inflate runs already set. Signed-off-by: Parker Loflin <parker@cronstable.dev>
reset_match_history() fills the hash table with wmemset(). wchar_t is 4 bytes on Linux and macOS, and wmemset() is not vectorized in every C library: on an Apple M4 it takes 4.0 us to fill a 64 KB level 2 or 3 hash table, where memset() takes 0.33 us. At the start of a stream the fill value is 0, so clear the table with memset(), and use a plain loop for other fill values. The compressed output does not change. On an Apple M4, stateless level 0 compression of 1 KB inputs goes from 686 to 813 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
encode_deflate_icf_aarch64 stores the bit buffer to memory after every symbol and loads it back, which puts store-to-load forwarding on the loop-carried dependency. Keep the bit buffer in registers and store it on exit. Combine the length code, distance code, and distance extra bits before shifting them into the bit buffer, so only one shift depends on the bit count. The compressed output does not change. On an Apple M4, deflate on the Silesia corpus goes from 381 to 460 MB/s at level 1 and from 348 to 409 MB/s at level 2. Signed-off-by: Parker Loflin <parker@cronstable.dev>
isal_deflate_icf_body_hash_hist_aarch64 writes each deflate_icf entry with three read-modify-write sequences, branches separately on the hash distance and on the match length, and compares matches 8 bytes at a time with each load address depending on the previous comparison. Rewrite the body. Write each entry with one 32-bit store. Compare the first 8 bytes of every candidate inline: when the hash distance is out of range, compare the input with itself and force the length to 0, so each position has one data-dependent branch. Compare longer matches 16 bytes per iteration and count the matching bytes once at the end. Compute the distance code without branches. The hash function, the hash table updates, and the compressed output do not change. On an Apple M4, deflate on the Silesia corpus goes from 460 to 578 MB/s at level 1 and from 408 to 585 MB/s at level 2. Signed-off-by: Parker Loflin <parker@cronstable.dev>
isal_deflate_body_aarch64 stores the bit buffer to the stream after every symbol, branches separately on the hash distance and on the match length, and compares matches with compare_258_bytes, whose next load address depends on the previous comparison. Rewrite the body like the level 1 and 2 body. Keep the bit buffer in registers and store it on exit. Compare the first 8 bytes of every candidate inline, with one data-dependent branch per position, and compare longer matches 16 bytes per iteration. Compute the distance code for distances above 2 without branches. Remove compare_258_bytes, which has no other users. The hash function, the hash table updates, and the compressed output do not change. On an Apple M4, level 0 deflate on the Silesia corpus goes from 575 to 681 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
On CPUs with CRC32, the aarch64 level 3 body reaches gen_icf_map_h1_aarch64 through the gen_icf_map_lh1 dispatcher. The function hashes with CRC32C instead of the compute_hash() multiplicative hash of gen_icf_map_h1_base(), and branches on whether each position starts a match. Replace gen_icf_map_h1_aarch64 with a version that produces the same output as gen_icf_map_h1_base(), including the compute_hash() hash. Its loop has no data-dependent branches: for each position it builds both the literal and the match entry and selects one. Insert level 3 preset dictionaries with isal_deflate_hash_base(), which also hashes with compute_hash(), so that level 3 still finds dictionary matches. Level 3 output then matches the C implementation. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The aarch64 level 0, 1, and 2 bodies count the matching bytes among the first 8 (eor, rbit, clz, and a shift) before the branch that separates literals from matches. That branch mispredicts often, and it resolves only after the count is known. Branch on a 4-byte comparison of the input and the candidate, folded into the distance range check with ccmp, and count the matching bytes only on the match path. A match needs 4 equal bytes, so the branch goes the same way as before. The compressed output does not change. On an Apple M4, deflate on the Silesia corpus goes from 693 to 739 MB/s at level 0, from 582 to 609 MB/s at level 1, and from 590 to 616 MB/s at level 2. Signed-off-by: Parker Loflin <parker@cronstable.dev>
gen_icf_map_h1_aarch64 handles one position per iteration with about 35 scalar instructions, and instruction throughput limits it. On an Apple M4, each instruction added to the loop slows level 3 by about 1.5%. Handle four positions per iteration. NEON computes the four compute_hash() values, the distance codes, and the deflate_icf entries. The hash table updates, the candidate loads, and the match lengths stay scalar and in order, so each position sees the hash table update of the position before it. The scalar loop handles the last positions. Its compute_hash() uses a 32-bit multiply for the second product, which gives the same masked hash because the hash mask is less than 1 << 16. The compressed output does not change. Signed-off-by: Parker Loflin <parker@cronstable.dev>
set_long_icf_fg_aarch64 checks one match map entry per iteration for an 8-byte match, extends those matches 8 bytes at a time, and copies a 128-byte distance table to the stack with memcpy() on every call. Find the entries with 8-byte matches 8 entries at a time with NEON, extend the matches 16 bytes at a time, and read the distance table in place. The compressed output does not change. Signed-off-by: Parker Loflin <parker@cronstable.dev>
ptweezy
force-pushed
the
igzip-aarch64-speedups
branch
from
October 24, 2026 02:53
259af37 to
19e32fe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR speeds up aarch64 deflate at levels 0 to 3. The ICF encoder and the level 0 body keep their bit buffers in registers, and the level 0 to 2 bodies check each match candidate with one data-dependent branch and compare long matches 16 bytes at a time. At level 3,
gen_icf_map_h1_aarch64is rewritten to matchgen_icf_map_h1_base()and to handle four positions per iteration with NEON, andset_long_icf_fg_aarch64checks eight entries at a time. The first commit for review clears the hash table withmemset()instead ofwmemset(), which is slow on macOS.Where it runs
The
memset()commit changesigzip/igzip.con every architecture. The rest is aarch64 code. As on master, the dispatcher selects the level 0 to 2 bodies and the level 3 match map generator on CPUs with CRC32, and the ICF encoder (encode_df.S) andset_long_icf_fg_aarch64on every aarch64 CPU. The one dispatcher change makes level 3 insert preset dictionaries withisal_deflate_hash_base()on every aarch64 CPU, since the match map generator hashes withcompute_hash().Output
Levels 0 to 2 compress to the same bytes as master with #439 at every commit. On CPUs with CRC32, master's level 3 match map hashes with CRC32C. From "speed up aarch64 level 3 match map generation" on, it hashes with
compute_hash(), so level 3 output matches the C implementation (arch=noarch), and no later commit changes it. Stateless level 3 output of the Silesia files is 0.04% larger than master's (75,636,323 against 75,603,510 bytes).Dependencies
Stacked on #439 (the level 1 and 2 rewrite builds on #439's ICF buffer fix): the first three commits are #439's. Review starts at "igzip: reset the deflate hash table with memset". Through #439's tests, this PR conflicts with #448 in
igzip/igzip_rand_test.c.Results
Neoverse N2 runners, master → this PR in MB/s (run, harness): in-process A/B of the two libraries, median of 9 runs on three runners of the best of 11 rounds, on the Silesia files and 256 slices of the Silesia tar per size.
Against its parent, "speed up aarch64 level 3 match map generation" makes level 3 14% faster on the whole files (150 → 171 MB/s) and 2%, 8%, 12% and 14% faster on 1 KB, 4 KB, 16 KB and 64 KB slices.
Testing
tools/check_format.shpasses on every commit. Under qemu-user (GCC 13.3),make -f Makefile.unx host_cpu=aarch64 arch=aarch64 checkpasses at every commit.make -f Makefile.unx checkpasses at the tip on x86-64 Linux and witharch=noarch.tools/test_extended.shpasses on a Neoverse N2 runner (run).hist_bits, dictionaries, level buffer sizes and random buffer sizes; every input buffer ends at aPROT_NONEpage) compared each commit with master with igzip: fix full flush corruption and two other bugs #439 on 15,000 cases and the tip on 60,000: levels 0 to 2 are byte-identical at every commit, and level 3 until "speed up aarch64 level 3 match map generation". From that commit on, level 3 is byte-identical to thearch=noarchbuild. zlib inflates every finished raw deflate, gzip and zlib stream without a preset dictionary, and nothing is read past an input buffer.