Repository navigation
Conversation
When all buffered input has been compressed and the flush type is FULL_FLUSH, get_hist_size() returns 0, so isal_deflate() keeps no history. That is only safe after sync_flush() writes the sync marker: it then sets has_hist to IGZIP_NO_HIST, and the next call resets the match history. If the output buffer fills before sync_flush() writes the sync marker, the flush is still pending when isal_deflate() returns, and has_hist is still IGZIP_HIST. If the caller supplies more input before the flush completes, the next call compresses that input with the old hash table but without the history it refers to. At every level, match candidates can then point up to 32 KB before the start of the caller's input buffer, so the compressor reads memory outside the input. When those bytes happen to match, it emits a valid deflate stream that inflates to different data. Keep the history until the full flush resets has_hist. Callers that wait for each flush to complete before supplying more input get the same output as before. Signed-off-by: Parker Loflin <parker@cronstable.dev>
level_buf->icf_buf_avail_out holds the free space in the ICF buffer in bytes. init_new_icf_block() and compress_icf_map_g() store bytes, every reader treats the value as bytes, and the x86 level 1 and 2 functions subtract the bytes they write. update_state() in igzip_icf_base.c and the aarch64 level 1 and 2 body and finish functions store a count of deflate_icf entries instead, so every call that leaves the block open cuts the remaining space to a quarter. The block then ends early. In stateless compression this happens when the finish function runs after the body. In stateful compression it happens each time a call runs out of input, so smaller input buffers give more blocks. On dickens at level 1, 4 KB input buffers give 45.11% instead of the 45.01% of a single call. Store the space in bytes. Stateful level 1 and 2 compression of dickens in 4 KB, 16 KB, or 64 KB input buffers then gives the same size as a single call. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The aarch64 level 3 body, icf_body_hash1_fillgreedy_lazy(), finds matches with gen_icf_map_h1_base(), which hashes with compute_hash(). The isal_deflate_hash_lvl3 dispatcher selects isal_deflate_hash_aarch64(), which inserts the preset dictionary with a CRC32C hash, so level 3 misses most dictionary matches. With 8 KB of random data as both the dictionary and the input, level 3 output is 5239 bytes, against 366 bytes with the C implementation. Select isal_deflate_hash_base(), which the x86 dispatcher pairs with the same level 3 body. Signed-off-by: Parker Loflin <parker@cronstable.dev>
print_perf_line() picks the label from the deflate strategy, so the isal stateless, isal stateful, and zlib inflate runs all print the label of the deflate run before them, for example zlib_inflate for every inflate run after a zlib level. Use info->inflate_mode, which the inflate runs already set. Signed-off-by: Parker Loflin <parker@cronstable.dev>
compare_max_258_bytes and compare_aarch64 load 8 bytes from each string per iteration and clamp the match length only after the loads, so their last iteration can read up to 7 bytes past the end of the input. Use the 8-byte loads only while at least 8 bytes remain, and compare the remaining bytes one at a time. The tail loop at second_loop_start in isal_deflate_finish_aarch64 loads 4 bytes to encode each of the last 1 to 3 literals, which reads up to 3 bytes past the end of the input. Load a single byte instead. The compressed output does not change. Signed-off-by: Parker Loflin <parker@cronstable.dev>
reset_match_history() fills the hash table with wmemset(). wchar_t is 4 bytes on Linux and macOS, and wmemset() is not vectorized in every C library: on an Apple M4 it takes 4.0 us to fill a 64 KB level 2 or 3 hash table, where memset() takes 0.33 us. At the start of a stream the fill value is 0, so clear the table with memset(), and use a plain loop for other fill values. The compressed output does not change. On an Apple M4, stateless level 0 compression of 1 KB inputs goes from 686 to 813 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
decode_huffman_code_block_stateless_aarch64 checks the buffer bounds, refills the bit buffer, and saves the decoder state for every symbol. It writes literals one byte at a time and moves the distance base and extra bit count through NEON registers. It copies matches 4 bytes at a time, or 1 byte at a time when the source overlaps the destination. Add a fast loop that runs while at least 8 bytes of input and 320 bytes of output space remain, and keep the existing loop for the ends of the buffers. The fast loop keeps the decoder state in registers. It refills the bit buffer with one 8-byte load, which covers a whole length and distance pair. It looks up the next symbol before the refill completes and writes up to 3 packed literals with one store. It reads the distance extra bit count from the table entry, copies matches in 16- or 32-byte chunks, and expands distances below 16 with TBL. Invalid symbols and distances use the existing error paths, so the decoder returns the same error codes at the same bit position. On an Apple M4, isal_inflate_stateless() on the Silesia corpus goes from 531 to 1056 MB/s for data compressed at level 1, and from 605 to 1222 MB/s for data compressed by zlib at level 6. Signed-off-by: Parker Loflin <parker@cronstable.dev>
encode_deflate_icf_aarch64 stores the bit buffer to memory after every symbol and loads it back, which puts store-to-load forwarding on the loop-carried dependency. Keep the bit buffer in registers and store it on exit. Combine the length code, distance code, and distance extra bits before shifting them into the bit buffer, so only one shift depends on the bit count. The compressed output does not change. On an Apple M4, deflate on the Silesia corpus goes from 381 to 460 MB/s at level 1 and from 348 to 409 MB/s at level 2. Signed-off-by: Parker Loflin <parker@cronstable.dev>
isal_deflate_icf_body_hash_hist_aarch64 writes each deflate_icf entry with three read-modify-write sequences, branches separately on the hash distance and on the match length, and compares matches 8 bytes at a time with each load address depending on the previous comparison. Rewrite the body. Write each entry with one 32-bit store. Compare the first 8 bytes of every candidate inline: when the hash distance is out of range, compare the input with itself and force the length to 0, so each position has one data-dependent branch. Compare longer matches 16 bytes per iteration and count the matching bytes once at the end. Compute the distance code without branches. The hash function, the hash table updates, and the compressed output do not change. On an Apple M4, deflate on the Silesia corpus goes from 460 to 578 MB/s at level 1 and from 408 to 585 MB/s at level 2. Signed-off-by: Parker Loflin <parker@cronstable.dev>
isal_deflate_body_aarch64 stores the bit buffer to the stream after every symbol, branches separately on the hash distance and on the match length, and compares matches with compare_258_bytes, whose next load address depends on the previous comparison. Rewrite the body like the level 1 and 2 body. Keep the bit buffer in registers and store it on exit. Compare the first 8 bytes of every candidate inline, with one data-dependent branch per position, and compare longer matches 16 bytes per iteration. Compute the distance code for distances above 2 without branches. Remove compare_258_bytes, which has no other users. The hash function, the hash table updates, and the compressed output do not change. On an Apple M4, level 0 deflate on the Silesia corpus goes from 575 to 681 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The aarch64 level 3 body, icf_body_hash1_fillgreedy_lazy(), calls gen_icf_map_h1_base() directly, so nothing uses gen_icf_map_h1_aarch64. That function also hashes with CRC32C instead of the compute_hash() that gen_icf_map_h1_base() and isal_deflate_hash_base() use, and on an Apple M4 it is no faster than the C function. Replace gen_icf_map_h1_aarch64 with a version that produces the same output as gen_icf_map_h1_base(), including the compute_hash() multiplicative hash. Its loop has no data-dependent branches: for each position it builds both the literal and the match entry and selects one. Select icf_body_lazyhash1_fillgreedy_greedy() for level 3, which reaches it through the gen_icf_map_lh1 dispatcher. The compressed output does not change. On an Apple M4, level 3 deflate on the Silesia corpus goes from 238 to 315 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
In the aarch64 inflate fast loop, the branch that separates literals from lengths mispredicts often, because whether a table entry ends in a length depends on the data. After the branch predicts a literal for an entry that ends in a length, the length path starts by loading the distance table entry. The branch itself resolves only after six instructions that depend on the table entry. Look up the distance table entry together with the next lit/len entry, before the branch: a distance code starts where the lit/len code ends. Test the last packed symbol with a shift and a mask instead of extracting it and comparing it with 256, so the branch resolves after four dependent instructions. On an Apple M4, isal_inflate_stateless() on the Silesia corpus goes from 1254 to 1383 MB/s for data compressed by zlib at level 6, and from 1060 to 1161 MB/s for data compressed by isa-l at level 1. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The aarch64 inflate fast loop runs only while at least 320 bytes of output space remain, enough for 2 literals, a 258-byte match, and the over-copy of the match. The per-symbol loop decodes the rest, which is the last 320 bytes of every stream and a large part of small streams. Within 320 bytes of the end of the output, check each symbol before decoding it. A literal entry writes at most 4 bytes, and a match of length len writes at most len + 33 bytes. Stay in the fast loop while the symbol fits and at least 64 bytes of space remain, and leave codes longer than ISAL_DECODE_LONG_BITS to the per-symbol loop. Both loops decode the same way, so the output and the return codes do not change. On an Apple M4, isal_inflate_stateless() of 1 KB, 4 KB, and 16 KB streams compressed at level 1 gets 5.8%, 3.1%, and 1.4% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The aarch64 level 0, 1, and 2 bodies count the matching bytes among the first 8 (eor, rbit, clz, and a shift) before the branch that separates literals from matches. That branch mispredicts often, and it resolves only after the count is known. Branch on a 4-byte comparison of the input and the candidate, folded into the distance range check with ccmp, and count the matching bytes only on the match path. A match needs 4 equal bytes, so the branch goes the same way as before. The compressed output does not change. On an Apple M4, deflate on the Silesia corpus goes from 693 to 739 MB/s at level 0, from 582 to 609 MB/s at level 1, and from 590 to 616 MB/s at level 2. Signed-off-by: Parker Loflin <parker@cronstable.dev>
build_huff_tree() repeatedly removes the two smallest keys from a binary heap and inserts the key of their parent. Each removal sifts a key down the heap with an unpredictable branch at each level. When level 1 compresses a 4 KB input on an Apple M4, build_heap() and build_huff_tree() take 28% of the time. Replace the C build_heap() and build_huff_tree() with a construction that gives the same tree. Sort the leaf keys with a radix sort on the frequency, and keep the keys of new nodes in the order they are made, which is ascending order of frequency. New nodes get decreasing ids, so among new nodes with the same frequency, the newest one has the smallest key. The smallest remaining key is then either the smallest leaf or the newest new node with the lowest frequency. The sort uses the tree nodes as scratch space, before they hold the tree. aarch64, ppc64le, riscv64, and base builds use the C functions, and x86 keeps its assembly versions. The compressed output does not change: for 900,000 random histograms, many of them with equal frequencies, the new functions build the same tree node for node. On an Apple M4, stateless level 1 and 2 compression of 1 KB, 4 KB, 16 KB, and 64 KB inputs gets 35%, 23%, 8%, and 2% faster, and level 3 gets 29%, 16%, 4%, and 2% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The Huffman code generation for each deflate block has loops with data-dependent branches: init_heap32() and the other init_heap functions skip zero frequencies, fix_code_lens() handles leaves and internal nodes differently, and set_huff_codes() skips unused symbols. The init_heap functions also clear the whole 6.9 KB heap_tree for each of the three trees of a block, even for the 19 code length symbols. Store every key and advance past the nonzero ones, handle both kinds of tree node with the same stores, and assign codes to unused symbols too, which get the code 0 they already have. Clear only the tree nodes that build_huff_tree() fills. The compressed output does not change on aarch64 or x86. On an Apple M4, stateless level 1 compression of 1 KB and 4 KB inputs gets 6% and 3% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
gen_icf_map_h1_aarch64 handles one position per iteration with about 35 scalar instructions, and instruction throughput limits it. On an Apple M4, each instruction added to the loop slows level 3 by about 1.5%. Handle four positions per iteration. NEON computes the four compute_hash() values, the distance codes, and the deflate_icf entries. The hash table updates, the candidate loads, and the match lengths stay scalar and in order, so each position sees the hash table update of the position before it. The scalar loop handles the last positions. Its compute_hash() uses a 32-bit multiply for the second product, which gives the same masked hash because the hash mask is less than 1 << 16. The compressed output does not change. On an Apple M4, level 3 deflate on the Silesia corpus goes from 318 to 410 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
set_long_icf_fg_aarch64 checks one match map entry per iteration for an 8-byte match, extends those matches 8 bytes at a time, and copies a 128-byte distance table to the stack with memcpy() on every call. Find the entries with 8-byte matches 8 entries at a time with NEON, extend the matches 16 bytes at a time, and read the distance table in place. Remove the compare_aarch64 macro, which has no other users. The compressed output does not change. On an Apple M4, level 3 deflate on the Silesia corpus goes from 410 to 420 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
On x86, build_huff_tree() comes from proc_heap.asm, which keeps the keys in a binary heap. Each insertion and removal takes a data-dependent branch at every level of the heap. The assembly also writes only the child id of each tree node, so the init_heap functions clear the tree nodes before each tree. Use the C build_huff_tree() from proc_heap_base.c on every architecture. It gives the same tree and writes each node in full, so remove build_heap() and the clearing of the tree nodes, and remove proc_heap.asm and heap_macros.asm. The compressed output does not change. On an AMD Ryzen 7 9800X3D, stateless level 1 compression of 1 KB, 4 KB, and 16 KB inputs gets 41%, 30%, and 12% faster, and level 3 compression 36%, 22%, and 8%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
isal_adler32() uses adler32_avx2_4 on every CPU with AVX2. It adds 16 bytes per iteration into one pair of accumulators, so each iteration waits for the previous one, and it runs at about 19 GB/s on an AMD Ryzen 7 9800X3D. Deflate and inflate with a zlib wrapper call it for all of their data. Add adler32_avx512 for CPUs with the AVX-512 extensions that come with VNNI. Each iteration takes 256 bytes in four 64 byte blocks: vpdpbusd multiplies each block by the weights 64 to 1 into its own accumulator, and vpsadbw sums the bytes, which give the 64 * a term of each block from the sums of the blocks before it. Each chunk of at most 5504 bytes is reduced modulo 65521 with 64-bit arithmetic, and the last 0 to 63 bytes use a masked load. The function uses only zmm16 to zmm31, so it saves no registers on Windows. On the same CPU, adler32 of 4 KB to 256 KB buffers in cache goes from 19 GB/s to 225 to 270 GB/s, and of 64 byte buffers from 6 to 17 GB/s. Stateless inflate of zlib streams of 4 KB and up gets 3% to 6% faster, and level 1 deflate with a zlib wrapper of 16 KB and up 3% to 4% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
On an AMD Ryzen 7 9800X3D, crc32_iscsi_by16_10 runs at about 82 GB/s for buffers of 4 KB and up. Its loop is limited by vpclmulqdq, which issues one zmm instruction every two cycles there, while the crc32 instruction, which the CPU runs twice per cycle, sits idle. Add crc32_iscsi_by16_10_fused, built from the same source with FUSED_CRC32. For buffers of at least 4 KB, it splits each 3584 byte chunk into 1792 bytes for the folding loop and eight 224 byte streams for crc32, and runs 32 crc32 per fold iteration. The folding loop skips over the streams with the constants for the longer distance. The stream crcs are shifted into place with two vpclmulqdq, combined, and xored into the second block after the streams like an initial crc, so the folding of one block hides their latency. Dispatch to it on AMD CPUs of family 1Ah and later where crc32_iscsi_by16_10 would be used. Other CPUs keep crc32_iscsi_by16_10: a CPU that runs one crc32 per cycle, like current Intel CPUs, could not keep up with 32 crc32 per iteration. On the same CPU, crc32_iscsi goes from 82 GB/s to 130 GB/s for 4 KB buffers, to 138-147 GB/s for 16 KB to 256 KB, to 137 GB/s for 1 MB, and from 79 to 92 GB/s for 4 MB. Shorter buffers do not change. Signed-off-by: Parker Loflin <parker@cronstable.dev>
This was referenced Oct 19, 2026
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is stacked on #440, which builds on #437 and #439. Their 11 commits come first, so review starts at "igzip: resolve aarch64 inflate length branches sooner". The compressed output is byte-identical to #440 on every architecture I tested.
Results
I timed the stateless API on an Apple M4 against #440 at 806d0db. I alternated the two builds and kept each build's best run.
For small inputs, I timed 256 slices of the Silesia tar per size, best of three rounds:
Changes
Items follow the seven new commits in order; each commit message has its own measurements.
build_huff_tree()builds the tree without a heap. It radix-sorts the leaves and merges them with the new nodes, which it creates in ascending order of frequency. For 900,000 random histograms, many with equal frequencies, the tree matches the heap version node for node.init_heap*(),fix_code_lens(), andset_huff_codes()lose their data-dependent branches, andinit_heap*()clears only the tree nodes thatbuild_huff_tree()fills.gen_icf_map_h1_aarch64handles four positions per iteration, using NEON for the hashes and distance codes and in-order scalar code for hash table updates and match lengths.set_long_icf_fg_aarch64finds 8-byte matches eight entries at a time with NEON and extends them 16 bytes at a time.igzip/proc_heap_base.carch=noarch(x86 usesproc_heap.asm)igzip/huff_codes.cI did not test ppc64le or riscv64, but
arch=noarchbuilds run the same C code.Testing
arch=noarchbuilds of both Linux targets:total_out,avail_in, and output for valid and corrupted streams, with buffers ending at aPROT_NONEpage.PROT_NONEpage.igzip_rand_test,igzip_wrapper_hdr_test,checksum32_funcs_test, andigzip_inflate_teston macOS.tools/check_format.shpasses, andtools/test_checks.shandtools/test_extended.shpass on Linux aarch64.