Repository navigation
Conversation
decode_huffman_code_block_stateless_aarch64 checks the buffer bounds, refills the bit buffer, and saves the decoder state for every symbol. It writes literals one byte at a time and moves the distance base and extra bit count through NEON registers. It copies matches 4 bytes at a time, or 1 byte at a time when the source overlaps the destination. Add a fast loop that runs while at least 8 bytes of input and 320 bytes of output space remain, and keep the existing loop for the ends of the buffers. The fast loop keeps the decoder state in registers. It refills the bit buffer with one 8-byte load, which covers a whole length and distance pair. It looks up the next symbol before the refill completes and writes up to 3 packed literals with one store. It reads the distance extra bit count from the table entry, copies matches in 16- or 32-byte chunks, and expands distances below 16 with TBL. Invalid symbols and distances use the existing error paths, so the decoder returns the same error codes at the same bit position. On an Apple M4, isal_inflate_stateless() on the Silesia corpus goes from 531 to 1056 MB/s for data compressed at level 1, and from 605 to 1222 MB/s for data compressed by zlib at level 6. Signed-off-by: Parker Loflin <parker@cronstable.dev>
In the aarch64 inflate fast loop, the branch that separates literals from lengths mispredicts often, because whether a table entry ends in a length depends on the data. After the branch predicts a literal for an entry that ends in a length, the length path starts by loading the distance table entry. The branch itself resolves only after six instructions that depend on the table entry. Look up the distance table entry together with the next lit/len entry, before the branch: a distance code starts where the lit/len code ends. Test the last packed symbol with a shift and a mask instead of extracting it and comparing it with 256, so the branch resolves after four dependent instructions. On an Apple M4, isal_inflate_stateless() on the Silesia corpus goes from 1254 to 1383 MB/s for data compressed by zlib at level 6, and from 1060 to 1161 MB/s for data compressed by isa-l at level 1. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The aarch64 inflate fast loop runs only while at least 320 bytes of output space remain, enough for 2 literals, a 258-byte match, and the over-copy of the match. The per-symbol loop decodes the rest, which is the last 320 bytes of every stream and a large part of small streams. Within 320 bytes of the end of the output, check each symbol before decoding it. A literal entry writes at most 4 bytes, and a match of length len writes at most len + 33 bytes. Stay in the fast loop while the symbol fits and at least 64 bytes of space remain, and leave codes longer than ISAL_DECODE_LONG_BITS to the per-symbol loop. Both loops decode the same way, so the output and the return codes do not change. On an Apple M4, isal_inflate_stateless() of 1 KB, 4 KB, and 16 KB streams compressed at level 1 gets 5.8%, 3.1%, and 1.4% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The main loop of decode_huffman_code_block_stateless_01 and _04 takes the index of the next distance lookup from read_in after the refill, so the lookup waits for the refill. Take the index before the refill, as the loop already does for the next lit/len lookup. read_in_length can be as low as 9 at that point, but the last refill loaded all 64 bits of read_in and at most 48 of them have been consumed since: 15 + 13 for a distance code and its extra bits, and 15 + 5 for a lit/len code and its extra bits. The low 16 bits of read_in already hold the stream bits that the refill would put there, so the 10-bit index, and with it the output, does not change. On GitHub-hosted runners with AMD Zen 3, Zen 4 and Zen 5 and Intel Ice Lake, Emerald Rapids and Granite Rapids CPUs, inflate gets 2.4% to 6.2% faster on the Silesia files compressed by zlib -6, and 1.0% to 4.6% faster on 64 KB pieces of them compressed at isa-l level 1. Signed-off-by: Parker Loflin <parker@cronstable.dev>
Before it decodes a dynamic block, inflate decodes the code lengths in the block header and builds the lit/len and distance decode tables, which on an AMD Ryzen 7 9800X3D takes about half the time of inflating a 1 KB slice of Silesia. It decodes the code lengths through the bit buffer in inflate_state, which every store can alias, branches on the length of each of the 256 literals, and compares every pair of long codes to find those that share their first 12 bits. Rewrite setup_dynamic_header() and the table builders: - Decode the code lengths into a byte array with the bit buffer in local variables and a 7 bit table, and list the literals of each code length as they are decoded. - Give the literals of each length consecutive codes from those lists, then expand the length symbols, which leaves every expanded code sorted by length without a separate sort. - Group the long codes by their first 12 bits in one pass. - Build the code length code table by doubling it once per length. The tables decode every input the same as master's. The long code table can be laid out differently, since master can make the subtable of the codes that start with twelve 1 bits bigger than they need. Header results and table decoding match master for the headers of Silesia blocks and 400,000 corrupted, truncated and crafted ones. On the same CPU, setting up the tables takes 33% to 39% fewer cycles for the headers of 1 KB to 64 KB blocks, and inflate of 1 KB, 4 KB and 16 KB slices of Silesia gets 21%, 10% and 5% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
When inflate decodes the final block of a stream, it leaves pairs and triples of symbols out of the lit/len table when little input is left, because they take longer to add than they save: it builds single symbols for up to 2 KB of input, adds pairs for up to 4 KB, and adds triples above that. With the tables faster to build, single symbols are still best for 1 KB slices of Silesia, which compress to about 500 bytes, but on an AMD Ryzen 7 9800X3D pairs pay off from about 1 KB of input, and triples only from about 8 KB. Build single symbols for up to 1 KB of input, pairs for up to 8 KB, and triples above that. The decompressed data does not change, but inflate decodes pairs and triples as a whole. When the output buffer fills up, the input consumed can differ from master's by a byte or two, and when the input is corrupted or ends too soon, inflate can stop one literal earlier. Continuing a valid stream gives the same result, and such differences also occur on master between inputs of different sizes. For final blocks of 4 KB to 8 KB of input, a stateless inflate into a too-small output buffer that master can end with ISAL_INVALID_LOOKBACK returns ISAL_OUT_OVERFLOW. On the same CPU, inflate of 3 KB, 4 KB and 8 KB slices of Silesia gets 1% to 3% faster, and of 2 KB slices up to 2% slower. 1 KB slices and slices of 16 KB and more stay within about 1%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The inflate decode tables keep codes longer than the short table index in long_code_lookup, which has 80 entries for the distance code and 1264 for the lit/len code. igzip_lib.h sizes it for the codes, entries that repeat codes shorter than the longest code of their group, and entries that no code uses, which only incomplete codes have. setup_dynamic_header() only rejects oversubscribed codes, so a header can define an incomplete code, and the bound on unused entries is too small: it allows 21 for the distance code, but a 15-bit code alone in the last group leaves 31 entries unused. An incomplete distance code can then need 86 entries: 30 codes, 25 repeats, and 31 unused entries. make_inflate_huff_code_dist() writes them without a check: it clears up to 12 bytes past long_code_lookup, which are the block_state, dict_length, and bfinal fields of inflate_state. master's make_inflate_huff_code_dist() has the same problem. Both table builders add up the space their long codes need before writing them, so return ISAL_INVALID_BLOCK if it is more than long_code_lookup holds. Complete codes always fit: a complete distance code needs at most 48 entries. zlib rejects incomplete codes as invalid, except a lit/len or distance code of a single code of length 1 and an empty distance code, none of which has long codes. The lit/len table gets the same check, since its size is worked out the same way, though no lit/len code is known that needs more than it holds. Signed-off-by: Parker Loflin <parker@cronstable.dev>
A distance decode table entry holds the symbol, the number of its extra bits and the length of its code. The decoders shift read_in by the code length, read the extra bits and shift read_in again before they can look up the next lit/len code, so both shifts are on the dependency chain from one lit/len lookup to the next. Store the length of the code plus its extra bits in bits 4:0 of the entry, the symbol in bits 10:6 and the code length in bits 14:11. A decoder then consumes the code and its extra bits with one shift by the entry itself, since shrx and lsr use only the low 6 bits of the count, and reads the extra bits off the critical path. The comment above DIST_TOTAL_LEN_MASK describes the invalid and subtable entries. Update the table builder, the C decoder, the x86 decoder and both loops of the aarch64 decoder, and regenerate static_inflate.h. On a Ryzen 7 9800X3D (Zen 5), inflate of isa-l level 1 streams of the Silesia files and of 64 KB pieces of them gets about 2% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
A lit/len decode table entry holds the length of its codes in bits 31:28, so the decoders shift it down before they can shift read_in, and the x86 decoder learns whether the entry ends with a length only after shifting out its literals. Both are on the main loop's critical paths. Hold the length of the codes in bits 4:0 of a short_code_lookup entry, so that decoders shift read_in by the entry itself, and the number of literals in bits 7:6, which is 0 when the entry ends with a length or end of block symbol. The comment above LARGE_CODE_LEN_MASK describes the format. The x86 main loop then tells literals from a length with one test of the count. Update the table builder, the C, x86 and aarch64 decoders, and regenerate static_inflate.h. Valid streams decode to the same output, but stop points can move: isal_inflate_stateless() can return up to two more literals from truncated input, isal_inflate() can leave avail_in a few bytes different, and errors in corrupted streams can be found a few bytes earlier or later. On a Ryzen 7 9800X3D (Zen 5), inflate of isa-l level 1 streams gets 12% to 13% faster for whole Silesia files, about 11% for 64 KB pieces and 5% to 6.5% for 4 KB pieces. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The main loop of the x86 inflate decoder stops OUT_BUFFER_SLOP (274) bytes before the end of the output buffer and leaves the rest to end_loop_block, which decodes a symbol at a time and copies matches with rep movsb. When the output buffer is the size of the output, that is a quarter of a 1 KB stream. Within OUT_BUFFER_SLOP of the end, decode the next lookup in the main loop if its output fits: 8 bytes for literals, and 1 + the length + 2 * COPY_SIZE bytes for a length, which covers the literal before it and the stores of both copy loops. Only the last bytes, and codes longer than the lookup, go to end_loop_block. Errors in the last 274 bytes of the output then take the main loop's error exits, whose total_out, avail_in and avail_out counts intel#446 fixes. Errors further from the end already take them, and avail_out wraps around in a few of those cases; near the end it wraps in most. Together with the next commit, inflate of 1 KB zlib -6 streams into an output buffer of their size gets 11% to 15% faster on GitHub-hosted x86 runners, and of 4 KB streams 4% to 7% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
Start decode_huffman_code_block_stateless_01 and _04 on a cache line, so that where the main loop falls within its cache lines does not depend on the code linked before the decoder. The effect depends on the CPU. On GitHub-hosted runners, this commit and the previous one change inflate into an output buffer at least 1 KB larger than the output, where the previous commit's code never runs, by +6.4% for whole Silesia files compressed by zlib -6 on AMD EPYC 9V74 (Zen 4), and by -0.7% to +1.0% on Zen 3, Zen 5, Ice Lake, Emerald Rapids and Granite Rapids. On a Ryzen 7 9800X3D (Zen 5), the alignment without the previous commit changes inflate of whole Silesia files by -1.2% to -0.2% and of 1 KB to 64 KB pieces by -0.2% to +2.1%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
ptweezy
force-pushed
the
igzip-inflate-one-shift-tables
branch
from
October 21, 2026 21:27
1d85edb to
c399b7b
Compare
ptweezy
marked this pull request as draft
October 21, 2026 21:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR changes the inflate decode table entries so that a decoder consumes a lit/len code, or a distance code with its extra bits, by shifting the bit buffer by the entry itself. That shortens the lit/len lookup chain that limits the x86 main loop. The third commit keeps that loop running into the last 274 bytes of an exact-size output buffer; the fourth aligns the x86 decoders to 64 bytes.
Code and builds
The first two commits change the table builders (all builds), the C decoder (
arch=noarch, riscv64), the x86_01and_04decoders and both aarch64 loops.igzip/static_inflate.h(+1896/-1896 of +2350/-2167) is generator output, so reviewers can skip it. The last two commits are x86 only.Output
Output does not change on valid streams, but stop points can:
isal_inflate_stateless()can return up to two more literals from truncated input,isal_inflate()can leaveavail_ina few bytes different, and errors in corrupted streams can come a few bytes earlier or later. With the third commit, an error in the last 274 bytes of the output takes the main loop's error exits, which miscounttotal_out,avail_inandavail_outuntil #446.Dependencies
Stacked on #465, whose aarch64 fast loop this PR updates; #456, whose early distance lookup the x86 loop keeps; and #464, whose table builders it changes. The last four commits are for review; the branch holds #465's commits and copies of #456's and #464's. #446 fixes those counts, and #458 fixes master's held-literal look-back check, which this stack also has. Through #464, it conflicts with #447 in
igzip/igzip_inflate.c.Results
GitHub-hosted runners, zlib -6 streams, MB/s: in-process A/B, median over all runs (two per runner) of the best of 11 rounds (harness, runs 1, 2, 3). In parentheses: the median over runs of the change in median-round throughput against the build named in the header. The 1 KB column comes from separate A/B pairs in the same jobs.
The last two commits are x86 only, so both Neoverse N2 percentages compare the first two commits with the base. On N2 this PR's gain is in #465's fast loop, which also gives most of the Silesia gain over master. Into a larger output buffer, where the third commit's code does not run, the last two commits change Silesia files by +6.4% on Zen 4 and by -0.7% to +1.0% on the other x86 CPUs.
Testing
make -f Makefile.unx checkpasses at each commit on x86-64 and, at the last, with_01forced,arch=noarch, aarch64 under qemu and mingw-w64 natively on Windows (Zen 5).tools/check_format.shpasses at each commit, andstatic_inflate.hregenerates (-DNO_STATIC_INFLATE_H) to the committed file._01,_04and aarch64 decoders return the same codes on valid, corrupted and truncated streams, stateless and stateful; outputs differ only at the stop points above or in the counts igzip: fix x86 inflate counts when the main loop finds an error #446 fixes. The C code is clean under ASan and UBSan, apart from master'smemcpy(NULL, 0)nonnull reports._01and_04match the C decoder in return code, counts and output on 167,188 stateless inflates ending at aPROT_NONEpage, held-literal streams pass with every decoder, and the near-end check holds with igzip: copy inflate matches 32 bytes at a time with AVX2 #457's 32-byte copies.