Repository navigation
Conversation
Before it decodes a dynamic block, inflate decodes the code lengths in the block header and builds the lit/len and distance decode tables, which on an AMD Ryzen 7 9800X3D takes about half the time of inflating a 1 KB slice of Silesia. It decodes the code lengths through the bit buffer in inflate_state, which every store can alias, branches on the length of each of the 256 literals, and compares every pair of long codes to find those that share their first 12 bits. Rewrite setup_dynamic_header() and the table builders: - Decode the code lengths into a byte array with the bit buffer in local variables and a 7 bit table, and list the literals of each code length as they are decoded. - Give the literals of each length consecutive codes from those lists, then expand the length symbols, which leaves every expanded code sorted by length without a separate sort. - Group the long codes by their first 12 bits in one pass. - Build the code length code table by doubling it once per length. The tables decode every input the same as master's. The long code table can be laid out differently, since master can make the subtable of the codes that start with twelve 1 bits bigger than they need. Header results and table decoding match master for the headers of Silesia blocks and 400,000 corrupted, truncated and crafted ones. On the same CPU, setting up the tables takes 33% to 39% fewer cycles for the headers of 1 KB to 64 KB blocks, and inflate of 1 KB, 4 KB and 16 KB slices of Silesia gets 21%, 10% and 5% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
When inflate decodes the final block of a stream, it leaves pairs and triples of symbols out of the lit/len table when little input is left, because they take longer to add than they save: it builds single symbols for up to 2 KB of input, adds pairs for up to 4 KB, and adds triples above that. With the tables faster to build, single symbols are still best for 1 KB slices of Silesia, which compress to about 500 bytes, but on an AMD Ryzen 7 9800X3D pairs pay off from about 1 KB of input, and triples only from about 8 KB. Build single symbols for up to 1 KB of input, pairs for up to 8 KB, and triples above that. The decompressed data does not change, but inflate decodes pairs and triples as a whole. When the output buffer fills up, the input consumed can differ from master's by a byte or two, and when the input is corrupted or ends too soon, inflate can stop one literal earlier. Continuing a valid stream gives the same result, and such differences also occur on master between inputs of different sizes. For final blocks of 4 KB to 8 KB of input, a stateless inflate into a too-small output buffer that master can end with ISAL_INVALID_LOOKBACK returns ISAL_OUT_OVERFLOW. On the same CPU, inflate of 3 KB, 4 KB and 8 KB slices of Silesia gets 1% to 3% faster, and of 2 KB slices up to 2% slower. 1 KB slices and slices of 16 KB and more stay within about 1%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The inflate decode tables keep codes longer than the short table index in long_code_lookup, which has 80 entries for the distance code and 1264 for the lit/len code. igzip_lib.h sizes it for the codes, entries that repeat codes shorter than the longest code of their group, and entries that no code uses, which only incomplete codes have. setup_dynamic_header() only rejects oversubscribed codes, so a header can define an incomplete code, and the bound on unused entries is too small: it allows 21 for the distance code, but a 15-bit code alone in the last group leaves 31 entries unused. An incomplete distance code can then need 86 entries: 30 codes, 25 repeats, and 31 unused entries. make_inflate_huff_code_dist() writes them without a check: it clears up to 12 bytes past long_code_lookup, which are the block_state, dict_length, and bfinal fields of inflate_state. master's make_inflate_huff_code_dist() has the same problem. Both table builders add up the space their long codes need before writing them, so return ISAL_INVALID_BLOCK if it is more than long_code_lookup holds. Complete codes always fit: a complete distance code needs at most 48 entries. zlib rejects incomplete codes as invalid, except a lit/len or distance code of a single code of length 1 and an empty distance code, none of which has long codes. The lit/len table gets the same check, since its size is worked out the same way, though no lit/len code is known that needs more than it holds. Signed-off-by: Parker Loflin <parker@cronstable.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For 1 KB inputs, decoding the dynamic block header and building the decode tables take about half the time of inflate. This PR cuts the per-block setup by 29% to 40%, uses pairs of symbols in final blocks from 1 KB instead of 2 KB and triples only from 8 KB instead of 4 KB, and rejects headers whose long codes overflow
long_code_lookup.Code and builds
C in
igzip/igzip_inflate.cthat every build runs for each dynamic block. The table format and the decoders do not change. With GCC 13 at-O2on x86-64,setup_dynamic_header()has a 9,024-byte stack frame, against 8,000 on master.Output
Decompressed data does not change. Pairs and triples decode as a whole, so with the new thresholds,
total_outandavail_incan differ from master by up to two bytes when the output buffer fills or the input is truncated or corrupted, and for final blocks of 4 KB to 8 KB, a stateless inflate into a too-small buffer can returnISAL_OUT_OVERFLOWwhere master can returnISAL_INVALID_LOOKBACK. Headers whose long codes overflowlong_code_lookupgetISAL_INVALID_BLOCK: an incomplete distance code can need 86 of the 80 entries, and master writes past the array.Conflicts
This PR conflicts with #447 in
igzip/igzip_inflate.c, where #447 adds the same check to master's builders.Results
Inflate of Silesia tar slices, master → this PR in MB/s, isa-l level 1 streams unless noted: in-process A/B, median over the runs of the best of 11 rounds; 256 slices per size on GitHub-hosted runners (run, Intel run; harnesses bdc0988, c7cde9c) and a Ryzen 7 9800X3D, 128 on an Apple M4; these builds and tests also include the deflate Huffman code changes, which touch no inflate code.
Header and table setup per block at the first commit (median of three runs of the median of 15 rounds; on the M4, through
isal_inflate_stateless()on header-only streams, including about 15 ns of fixed cost):Whole Silesia files inflate 0.3% to 1.1% faster on the runners and 0.4% to 0.8% faster on the M4, apart from code-placement effects of up to 2.7% (the Ryzen 7 9800X3D, within about 1% after relinking with padding, and xml's zlib -6 streams on the M4). With the aarch64 inflate fast path in both builds, this PR makes inflate of 1 KB, 4 KB, 16 KB and 64 KB slices on the M4 16%, 16%, 6% and 2% faster.
Testing
tools/check_format.shandmake -f Makefile.unx checkpass at every commit on x86-64 andarch=noarch, and on aarch64 on the M4 with the deflate changes included. At the tip,make checkalso passes with the_01decoder forced, for aarch64 under qemu-user, and for mingw-w64 x86_64 natively on Windows (Zen 5); the igzip tests pass under ASan and UBSan on the M4, andtools/test_extended.shpasses on Neoverse N2 runners.ISAL_OUT_OVERFLOWwherever master does.static_inflate.h, this PR's builders reproduce its 8 tables on x86-64, aarch64 andarch=noarch.