Repository navigation
Conversation
build_huff_tree() repeatedly removes the two smallest keys from a binary heap and inserts the key of their parent. Each removal sifts a key down the heap with an unpredictable branch at each level. When level 1 compresses a 4 KB input on an Apple M4, build_heap() and build_huff_tree() take 28% of the time. Replace the C build_heap() and build_huff_tree() with a construction that gives the same tree. Sort the leaf keys with a radix sort on the frequency, in the tree nodes before they hold the tree, and keep the keys of new nodes in the order they are made, which is ascending order of frequency. New nodes get decreasing ids, so among new nodes with the same frequency, the newest one has the smallest key. The smallest remaining key is then either the smallest leaf or the newest new node with the lowest frequency. aarch64, ppc64le, riscv64, and base builds use the C functions, and x86 keeps its assembly versions. The compressed output does not change: for 900,000 random histograms, many of them with equal frequencies, the new functions build the same tree node for node. The exception is a histogram given to isal_create_hufftables() or isal_create_hufftables_subset() whose counts add up to 2^48 or more. The keys hold the counts in 48 bits, so the sums of keys wrap around, and the two constructions then give different code lengths, both of them valid. On an Apple M4, with intel#440, stateless level 1 and 2 compression of 1 KB, 4 KB, 16 KB, and 64 KB inputs gets 35%, 23%, 8%, and 2% faster, and level 3 gets 29%, 16%, 4%, and 2% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The Huffman code generation for each deflate block has loops with data-dependent branches: init_heap32() and the other init_heap functions skip zero frequencies, fix_code_lens() handles leaves and internal nodes differently, and set_huff_codes() skips unused symbols. The init_heap functions also clear the whole 6.9 KB heap_tree for each of the three trees of a block, even for the 19 code length symbols. Store every key and advance past the nonzero ones, handle both kinds of tree node with the same stores, and assign codes to unused symbols too, which get the code 0 they already have. Clear only the tree nodes that build_huff_tree() fills. The compressed output does not change on aarch64 or x86. On an Apple M4, with intel#440, stateless level 1 compression of 1 KB and 4 KB inputs gets 6% and 3% faster. On an AMD Ryzen 7 9800X3D, stateless level 1 and level 3 compression of 1 KB inputs gets 5% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
On x86, build_huff_tree() comes from proc_heap.asm, which keeps the keys in a binary heap. Each insertion and removal takes a data-dependent branch at every level of the heap. The assembly also writes only the child id of each tree node, so the init_heap functions clear the tree nodes before each tree. Use the C build_huff_tree() from proc_heap_base.c on x86 too, and build proc_heap_base.c for every architecture. It gives the same tree and writes each node in full, so remove build_heap() and the clearing of the tree nodes, and remove proc_heap.asm and heap_macros.asm. The compressed output does not change, except, as on the other architectures, for histograms given to isal_create_hufftables() or isal_create_hufftables_subset() whose counts add up to 2^48 or more. On an AMD Ryzen 7 9800X3D, stateless level 1 compression of 1 KB, 4 KB, and 16 KB inputs gets 40%, 30%, and 12% faster, and level 3 compression 34%, 21%, and 8%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
create_hufftables_icf() takes about 15,000 cycles for a 1 KB block on an AMD Ryzen 7 9800X3D, two thirds of the time of compressing a 1 KB input at level 1. It handles all 286 lit/len symbols at each step: it gathers them into a heap, writes a length and gives a code to each, adds up the length of the block over all of them, and run length encodes the code lengths with a branch on each one. fix_code_lens() reads 8 byte tree nodes whose depth was just stored as 4 bytes, which the CPU cannot forward from the store. Rewrite it around the symbols that occur: - make_code_lens() gathers them, builds the tree with build_huff_tree(), and visits it in the order fix_code_lens() does, with the depths in a byte array. It then limits the code lengths the same way. - set_codes_by_sym() gives codes to these symbols only, and the length of the block adds up only these symbols. - rl_encode_lens() finds each run of code lengths with 8 byte compares. - write_dynamic_header() writes each code length code together with its extra bits, with the bit buffer in local variables. isal_create_hufftables() and isal_create_hufftables_subset() use rl_encode_lens() and write_dynamic_header() too, and the functions they replace are removed. sort_keys() sorts 32 keys or fewer with a branchless count of the smaller keys of each. The output does not change: create_hufftables_icf() gives the same tables, header bits, and return value as master for random and real histograms, and so do the two public functions when the counts add up to less than 2^48. On the same CPU, create_hufftables_icf() takes 30%, 25%, 21%, and 18% fewer cycles for blocks of 1 KB, 4 KB, 16 KB, and 1 MB slices of Silesia. Level 1 and 2 compression of 1 KB, 4 KB, and 16 KB slices gets 25%, 12%, and 5% faster, and level 3 compression 22%, 8%, and 2%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
make_code_lens() visits all 2n - 1 entries of the tree in the order fix_code_lens() does, with a branch-free loop that also lists the leaves in that order, which is only needed when the code lengths must be limited. Instead, walk the n - 1 internal nodes in increasing id order, from the root down, giving each pair of children its depth, so that the depth of each leaf ends up at its symbol, and count the code lengths from the symbols. Only when the tree is deeper than max_len, walk all 2n - 1 entries in that order and limit the lengths as before. Also gather the symbols with a 64 bit index, which saves the copies of 32 bit indexes, and expand the length codes in expand_hufftables_icf() by adding 1 << len per entry, two entries at a time. The output does not change. On an AMD Ryzen 7 9800X3D, level 1 and 2 compression of 1 KB slices of Silesia gets about 7% faster, and of 4 KB slices about 4%. Level 3 gets about 5% and 3% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
- sort_keys() sorts up to 32 keys by counting the smaller keys of each, n * n steps. When the frequencies fit in 8 bits, use one counting pass instead if it is cheaper (about 2 * n + 2^bits steps); with wider frequencies the 32-key cutoff stays. The radix sort expects the leaves in id order, as all callers write them. - build_huff_tree() searches for where the next frequency's nodes end each time it removes the last new node of a frequency. When no frequency is 0 and none reaches 2^48 / MAX_HISTHEAP_SIZE, as in every deflate block, record each end as the nodes are made instead: sums cannot wrap, so a new node never joins its frequency after one of its nodes has been removed. Other inputs, which only isal_create_hufftables() and isal_create_hufftables_subset() pass, keep the search. - Put new node keys in the slots of removed leaves, of which there are at least k + 2 when node k is made, instead of a 2 KB array. With gcc -O2, the deepest stack of build_huff_tree() goes from 2464 to 816 bytes. The tree does not change, also for counts that add up to 2^48 or more. Signed-off-by: Parker Loflin <parker@cronstable.dev>
rl_encode_lens() finds each run by comparing 8 lengths at a time with the first length of the run, so the start of each run waits for the loads and compares of the run before it, and it writes the codes of a run with branches on whether the length is 0 and how long the run is. Mark where the runs of each 64 lengths start in a bitmap, 8 lengths at a time, and go through the runs with the set bits. Each run then writes the length if it is not 0, and code 16, 17, or 18 or up to two more lengths, choosing them with masks. A code 16, 17 or 18 that ends a run is counted in registers. Only runs of more than 7 of the same nonzero length or of more than 138 zeros take a branch, which writes and counts their other codes 16 or 18. The output does not change. On an AMD Ryzen 7 9800X3D, level 1 and 2 compression of 1 KB slices of Silesia gets about 4% faster, and of 4 KB slices 1-2%. Level 3 gets 2-3% and about 1% faster. On Neoverse N2 it gets up to 1% slower. Signed-off-by: Parker Loflin <parker@cronstable.dev>
ptweezy
force-pushed
the
igzip-huff-code-lens-internal
branch
from
October 21, 2026 21:27
c3ffd16 to
cc45ab4
Compare
ptweezy
changed the title
This was referenced Oct 21, 2026
ptweezy
marked this pull request as draft
October 21, 2026 21:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
At levels 1-3, deflate builds Huffman codes and a dynamic header for every block, which takes a large share of the time for small inputs. Three commits remove branches from these steps. This is C code that every architecture runs, and
isal_create_hufftables()andisal_create_hufftables_subset()share the tree code.Stacked on #444; the commits for review are the last three.
Changes
make_code_lens()gets each leaf's depth by walking only the n - 1 internal nodes, from the root down. It walks all 2n - 1 entries only when the tree is deeper than the length limit.sort_keys()sorts 8-bit frequencies with one counting pass when that is cheaper than comparing every pair.build_huff_tree()records where each frequency's new nodes end as it makes them, instead of searching for the end. It does this when no frequency is 0 and none reaches 2^48 / 286, so sums cannot wrap, which every deflate block meets. Other inputs, which only the two public functions can pass, use the search and get the same tree.build_huff_tree()from 2464 to 816 bytes (gcc 13 -O2).rl_encode_lens()marks where the header's runs of code lengths start in a 64-bit bitmap, 8 lengths at a time, and picks each run's codes with masks. It branches only for runs of more than 7 equal nonzero lengths or more than 138 zeros.Output
Output matches #444's for every input. It matches master's except for histograms given to the two public functions whose counts add up to 2^48 or more (see #444).
Conflicts
Through #444, this PR conflicts with #452 in
igzip/Makefile.am,Makefile.nmakeandcmake/igzip.cmake.Results
Level 1 on GitHub runners, 1 KB and 4 KB Silesia slices (harness, runs 1, 2, 3), two runs per runner:
1 KB
4 KB
Commit 3 is slower on all three Neoverse N2 runners.
Testing
make -f Makefile.unx checkpasses at every commit on x86-64 and noarch, and at the head on aarch64 (qemu) and mingw-w64 x86_64 (native Windows).tools/check_format.shpasses.rl_encode_lens()matches igzip: build deflate Huffman codes faster #444 and master for all short sequences, single runs at every offset, and random mixes of runs: under ASan with exactly sized buffers on x86-64, and under qemu on aarch64.