Repository navigation
Conversation
build_huff_tree() repeatedly removes the two smallest keys from a binary heap and inserts the key of their parent. Each removal sifts a key down the heap with an unpredictable branch at each level. When level 1 compresses a 4 KB input on an Apple M4, build_heap() and build_huff_tree() take 28% of the time. Replace the C build_heap() and build_huff_tree() with a construction that gives the same tree. Sort the leaf keys with a radix sort on the frequency, and keep the keys of new nodes in the order they are made, which is ascending order of frequency. New nodes get decreasing ids, so among new nodes with the same frequency, the newest one has the smallest key. The smallest remaining key is then either the smallest leaf or the newest new node with the lowest frequency. The sort uses the tree nodes as scratch space, before they hold the tree. aarch64, ppc64le, riscv64, and base builds use the C functions, and x86 keeps its assembly versions. The compressed output does not change: for 900,000 random histograms, many of them with equal frequencies, the new functions build the same tree node for node. The exception is a histogram given to isal_create_hufftables() or isal_create_hufftables_subset() whose counts add up to 2^48 or more. The keys hold the counts in 48 bits, so the sums of keys wrap around, and the two constructions then give different code lengths, both of them valid. On an Apple M4, with the faster aarch64 deflate bodies of a separate patch series, stateless level 1 and 2 compression of 1 KB, 4 KB, 16 KB, and 64 KB inputs gets 35%, 23%, 8%, and 2% faster, and level 3 gets 29%, 16%, 4%, and 2% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The Huffman code generation for each deflate block has loops with data-dependent branches: init_heap32() and the other init_heap functions skip zero frequencies, fix_code_lens() handles leaves and internal nodes differently, and set_huff_codes() skips unused symbols. The init_heap functions also clear the whole 6.9 KB heap_tree for each of the three trees of a block, even for the 19 code length symbols. Store every key and advance past the nonzero ones, handle both kinds of tree node with the same stores, and assign codes to unused symbols too, which get the code 0 they already have. Clear only the tree nodes that build_huff_tree() fills. The compressed output does not change on aarch64 or x86. On an Apple M4, with the faster aarch64 deflate bodies of a separate patch series, stateless level 1 compression of 1 KB and 4 KB inputs gets 6% and 3% faster. On an AMD Ryzen 7 9800X3D, stateless level 1 and level 3 compression of 1 KB inputs gets 5% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
On x86, build_huff_tree() comes from proc_heap.asm, which keeps the keys in a binary heap. Each insertion and removal takes a data-dependent branch at every level of the heap. The assembly also writes only the child id of each tree node, so the init_heap functions clear the tree nodes before each tree. Use the C build_huff_tree() from proc_heap_base.c on every architecture. It gives the same tree and writes each node in full, so remove build_heap() and the clearing of the tree nodes, and remove proc_heap.asm and heap_macros.asm. The compressed output does not change, except, as on the other architectures, for histograms given to isal_create_hufftables() or isal_create_hufftables_subset() whose counts add up to 2^48 or more. On an AMD Ryzen 7 9800X3D, stateless level 1 compression of 1 KB, 4 KB, and 16 KB inputs gets 40%, 30%, and 12% faster, and level 3 compression 34%, 21%, and 8%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
Before it decodes a dynamic block, inflate decodes the code lengths in the block header and builds the lit/len and distance decode tables. On an AMD Ryzen 7 9800X3D this takes about 6,100 cycles for the header of a 1 KB block of Silesia, about half the time of inflating a 1 KB input, and 14,300 to 15,500 for a 64 KB block, depending on the compressor. The time goes to: - decoding the code lengths with the bit buffer in inflate_state, which every store to the code lengths can alias, and with a 10 bit table for the code length code, whose codes have at most 7 bits, - a branch on the code length of each of the 256 literals, and loads of counts and next codes right after they are stored, while giving the literals their codes and sorting them by length, - finding the long codes that share their first 12 bits by comparing every pair of long codes. Rewrite setup_dynamic_header() and the table builders: - Decode the code lengths into a byte array with the bit buffer in local variables and a 7 bit table, and add each literal to a list for its code length as it is decoded. - Give the literals of each length consecutive codes from those lists, then expand the length symbols, which leaves every expanded code sorted by length without a separate sort. - Group the long codes by their first 12 bits in one pass, numbering the groups in the short table entries that later point to them. - Build the code length code table by doubling it once per length, with the loop unrolled so the copies need no calls. The tables decode every input the same as before. The old code could count long codes of groups it had already handled as members of the group whose first 12 bits are all ones, which made that group's subtable bigger than its codes need. The new code sizes each subtable for its own codes, so the long code table can be laid out differently. For the headers of 3,035 dynamic blocks of Silesia slices compressed by zlib and by isa-l levels 1 and 3, 200,000 corrupted and truncated variants of them, and 200,000 headers with many long codes, the new code returns the same result, and its tables decode every input the same. Inflate gives the same output, return value, and remaining input as before for 40,000 valid and corrupted streams. The tables in static_inflate.h come out the same. On an AMD Ryzen 7 9800X3D, setting up the tables takes 38% fewer cycles for the headers of 1 KB blocks, 33% fewer for 4 KB and 16 KB blocks, and 37% fewer for 64 KB blocks. Inflate of 1 KB, 4 KB, 16 KB, and 64 KB slices of Silesia gets 21%, 10%, 5%, and 2% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
create_hufftables_icf() takes about 15,000 cycles for a 1 KB block on an AMD Ryzen 7 9800X3D, two thirds of the time of compressing a 1 KB input at level 1. It handles all 286 lit/len symbols at each step: it gathers them into a heap, writes a length and gives a code to each, adds up the length of the block over all of them, and run length encodes the code lengths with a branch on each one. fix_code_lens() reads 8 byte tree nodes whose depth was just stored as 4 bytes, which the CPU cannot forward from the store. Rewrite it around the symbols that occur: - make_code_lens() gathers them, builds the tree with build_huff_tree(), and visits it in the order fix_code_lens() does, with the depths in a byte array. It then limits the code lengths the same way. - set_codes_by_sym() gives codes to these symbols only, and the length of the block adds up only these symbols. - rl_encode_lens() finds each run of code lengths with 8 byte compares. - write_dynamic_header() writes each code length code together with its extra bits, with the bit buffer in local variables. isal_create_hufftables() and isal_create_hufftables_subset() now use rl_encode_lens() and write_dynamic_header() too, and the functions they replace are removed. sort_keys() sorts 32 keys or fewer by counting the smaller keys of each, which takes no branches, instead of with an insertion sort. The output does not change. create_hufftables_icf() gives the same tables, header bits, and return value for 500,000 random histograms, many with equal counts or counts that need their code lengths limited, and for the histograms of 2,454 blocks of Silesia. isal_create_hufftables() and isal_create_hufftables_subset() give the same tables for 300,000 random histograms, and deflate gives the same output for 40,000 random combinations of level, flush, wrapper, and buffer sizes. On an AMD Ryzen 7 9800X3D, create_hufftables_icf() takes 30%, 25%, 21%, and 18% fewer cycles for blocks of 1 KB, 4 KB, 16 KB, and 1 MB slices of Silesia. Level 1 and 2 compression of 1 KB, 4 KB, and 16 KB slices gets 25%, 12%, and 5% faster, and level 3 compression 22%, 8%, and 2%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
When inflate decodes the final block of a stream, it leaves pairs and triples of symbols out of the lit/len table when little input is left, because they take longer to add than they save: it builds single symbols for up to 2 KB of input, adds pairs for up to 4 KB, and adds triples above that. With the tables faster to build, single symbols are still best for 1 KB slices of Silesia, which compress to about 500 bytes, but on an AMD Ryzen 7 9800X3D pairs pay off from about 1 KB of input, and triples only from about 8 KB. Build single symbols for up to 1 KB of input, pairs for up to 8 KB, and triples above that. The decompressed data does not change, but inflate decodes pairs and triples as a whole. When the output buffer fills up, the input consumed can differ by a byte or two from before, and when the input is corrupted or ends too soon, inflate can stop one literal earlier. Continuing a valid stream gives the same result, and such differences could already happen between inputs of different sizes. On the same CPU, inflate of 3 KB slices of Silesia gets 1% to 3% faster, of 4 KB slices 2% to 3% faster, and of 8 KB slices about 1% faster, while inflate of 2 KB slices gets up to 2% slower. 1 KB slices and slices of 16 KB and more stay within about 1%. Single symbols for up to 1.5 KB of input lost most of the gain at 3 KB and did not measurably help at 2 KB. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The inflate decode tables keep codes longer than the short table index in long_code_lookup, which has 80 entries for the distance code and 1264 for the lit/len code. igzip_lib.h sizes it for the codes, entries that repeat codes shorter than the longest code of their group, and entries that no code uses, which only incomplete codes have. setup_dynamic_header() only rejects oversubscribed codes, so a header can define an incomplete code, and the bound on unused entries is too small: it allows 21 for the distance code, but a 15-bit code alone in the last group leaves 31 entries unused. An incomplete distance code can then need 86 entries: 30 codes, 25 repeats, and 31 unused entries. make_inflate_huff_code_dist() writes them without a check: it clears up to 12 bytes past long_code_lookup, which are the block_state, dict_length, and bfinal fields of inflate_state. master's make_inflate_huff_code_dist() has the same problem. Both table builders add up the space their long codes need before writing them, so return ISAL_INVALID_BLOCK if it is more than long_code_lookup holds. Complete codes always fit: a complete distance code needs at most 48 entries. zlib rejects incomplete codes as invalid, except a lit/len or distance code of a single code of length 1 and an empty distance code, none of which has long codes. The lit/len table gets the same check, since its size was worked out the same way, though no lit/len code was found that needs more than it holds. Signed-off-by: Parker Loflin <parker@cronstable.dev>
rl_encode_lens() finds each run by comparing 8 lengths at a time with the first length of the run, so the start of each run waits for the loads and compares of the run before it, and it writes the codes of a run with branches on whether the length is 0 and how long the run is. Mark where the runs of each 64 lengths start in a bitmap, 8 lengths at a time, and go through the runs with the set bits. Each run then writes the length if it is not 0, and code 16, 17, or 18 or up to two more lengths, choosing them with masks, and counts code 16, 17, and 18 in registers. Only runs that need more than one code 16 or code 18 take a branch. The output does not change. On an AMD Ryzen 7 9800X3D, level 1 and 2 compression of 1 KB slices of Silesia gets about 4% faster, and of 4 KB slices 1-2%. Level 3 gets 2-3% and about 1% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
Contributor
Author
|
Closing: this change is in #461 as "igzip: run length encode code lengths with fewer branches". |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR speeds up
rl_encode_lens(), which run length encodes the code lengths for the dynamic block header.This PR is stacked on #444, which adds
rl_encode_lens(). The commit for review is the last one, "igzip: run length encode code lengths with fewer branches".Results
I measured on an AMD Ryzen 7 9800X3D (Zen 5) under WSL2. Each cell is MB/s of input, the median over three runs of the best of 15 rounds, where each run alternates the libraries round by round in one process and compresses 256 slices of each size from silesia.tar:
Against #444, that is 4-5% at levels 1 and 2 and 3% at level 3 for 1 KB slices, and 1-2% for 4 KB slices. With every library relinked behind 0, 704 and 1408 bytes of padding, which moves all of its code, the gain stays at 2.8-4.3% for 1 KB slices and 1.2-2.2% for 4 KB slices at level 1. Whole Silesia files stay within 2% of #444.
Other CPUs
I timed #444 and this PR against each other on GitHub-hosted runners, with the workflow and harness in my fork (run, Intel and Zen 4 run, Zen 5, Ice Lake and Emerald Rapids run). The harness loads both builds'
libisal.sointo one process and alternates them round by round. Each cell is #444 → this PR in MB/s, the median over the runs on that CPU (two per runner) of the best of 11 rounds, and in parentheses the median change of the per-round medians. Slices are 256 pieces of the Silesia tar. The arm64 runners are all Neoverse N2.Master → #444:
On Neoverse N2, this PR is 0.2% to 1.0% slower than #444, consistently over the three runners.
Changes
rl_encode_lens()finds each run by comparing 8 lengths at a time with the first length of the run, so the start of each run waits for the loads and compares of the run before it, and it writes the codes of a run with branches on whether the length is 0 and how long the run is.This commit marks where the runs of each 64 lengths start in a bitmap, 8 lengths at a time, and goes through the runs with the set bits. Each run then writes the length if it is not 0, and code 16, 17, or 18 or up to two more lengths, choosing them with masks, and counts code 16, 17, and 18 in registers. Only runs that need more than one code 16 or code 18 take a branch.
Testing
make -f Makefile.unx checkpasses on x86-64, witharch=noarch, and for aarch64 under qemu.create_hufftables_icf()gives the same tables, header bits and return value as igzip: build deflate Huffman codes faster #444 for 302,454 histograms, captured from compression of 1 KB, 4 KB, 16 KB and 1 MB Silesia slices, plus random ones.isal_create_hufftables()andisal_create_hufftables_subset()give the same tables as igzip: build deflate Huffman codes faster #444 for 300,000 random histograms.rl_encode_lens()goes from 48 bytes to 56, which is not on the deepest stack path ofcreate_hufftables_icf().