Repository navigation
Conversation
When all buffered input has been compressed and the flush type is FULL_FLUSH, get_hist_size() returns 0, so isal_deflate() keeps no history. That is only safe after sync_flush() writes the sync marker: it then sets has_hist to IGZIP_NO_HIST, and the next call resets the match history. If the output buffer fills before sync_flush() writes the sync marker, the flush is still pending when isal_deflate() returns, and has_hist is still IGZIP_HIST. If the caller supplies more input before the flush completes, the next call compresses that input with the old hash table but without the history it refers to. At every level, match candidates can then point up to 32 KB before the start of the caller's input buffer, so the compressor reads memory outside the input. When those bytes happen to match, it emits a valid deflate stream that inflates to different data. Keep the history until the full flush resets has_hist. Callers that wait for each flush to complete before supplying more input get the same output as before. Add a test to igzip_rand_test that compresses with FULL_FLUSH and small output buffers at each level, and supplies more input while a flush is pending. Without the fix, it fails for most seeds, including the default one. Signed-off-by: Parker Loflin <parker@cronstable.dev>
level_buf->icf_buf_avail_out holds the free space in the ICF buffer in bytes. init_new_icf_block() and compress_icf_map_g() store bytes, every reader treats the value as bytes, and the x86 level 1 and 2 functions subtract the bytes they write. update_state() in igzip_icf_base.c and the aarch64 level 1 and 2 body and finish functions store a count of deflate_icf entries instead, so every call that leaves the block open cuts the remaining space to a quarter. The block then ends early. In stateless compression this happens when the finish function runs after the body. In stateful compression it happens each time a call runs out of input, so smaller input buffers give more blocks. On dickens at level 1, 4 KB input buffers give 45.11% instead of the 45.01% of a single call. Store the space in bytes. Stateful level 1 and 2 compression of dickens in 4 KB, 16 KB, or 64 KB input buffers then gives the same size as a single call. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The aarch64 level 3 body, icf_body_hash1_fillgreedy_lazy(), finds matches with gen_icf_map_h1_base(), which hashes with compute_hash(). The isal_deflate_hash_lvl3 dispatcher selects isal_deflate_hash_aarch64(), which inserts the preset dictionary with a CRC32C hash, so level 3 misses most dictionary matches. With 8 KB of random data as both the dictionary and the input, level 3 output is 5239 bytes, against 366 bytes with the C implementation. Select isal_deflate_hash_base(), which the x86 dispatcher pairs with the same level 3 body. Signed-off-by: Parker Loflin <parker@cronstable.dev>
print_perf_line() picks the label from the deflate strategy, so the isal stateless, isal stateful, and zlib inflate runs all print the label of the deflate run before them, for example zlib_inflate for every inflate run after a zlib level. Use info->inflate_mode, which the inflate runs already set. Signed-off-by: Parker Loflin <parker@cronstable.dev>
compare_max_258_bytes and compare_aarch64 load 8 bytes from each string per iteration and clamp the match length only after the loads, so their last iteration can read up to 7 bytes past the end of the input. Use the 8-byte loads only while at least 8 bytes remain, and compare the remaining bytes one at a time. The tail loop at second_loop_start in isal_deflate_finish_aarch64 loads 4 bytes to encode each of the last 1 to 3 literals, which reads up to 3 bytes past the end of the input. Load a single byte instead. The compressed output does not change. Signed-off-by: Parker Loflin <parker@cronstable.dev>
reset_match_history() fills the hash table with wmemset(). wchar_t is 4 bytes on Linux and macOS, and wmemset() is not vectorized in every C library: on an Apple M4 it takes 4.0 us to fill a 64 KB level 2 or 3 hash table, where memset() takes 0.33 us. At the start of a stream the fill value is 0, so clear the table with memset(), and use a plain loop for other fill values. The compressed output does not change. On an Apple M4, stateless level 0 compression of 1 KB inputs goes from 686 to 813 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
decode_huffman_code_block_stateless_aarch64 checks the buffer bounds, refills the bit buffer, and saves the decoder state for every symbol. It writes literals one byte at a time and moves the distance base and extra bit count through NEON registers. It copies matches 4 bytes at a time, or 1 byte at a time when the source overlaps the destination. Add a fast loop that runs while at least 8 bytes of input and 320 bytes of output space remain, and keep the existing loop for the ends of the buffers. The fast loop keeps the decoder state in registers. It refills the bit buffer with one 8-byte load, which covers a whole length and distance pair. It looks up the next symbol before the refill completes and writes up to 3 packed literals with one store. It reads the distance extra bit count from the table entry, copies matches in 16- or 32-byte chunks, and expands distances below 16 with TBL. Invalid symbols and distances use the existing error paths, so the decoder returns the same error codes at the same bit position. On an Apple M4, isal_inflate_stateless() on the Silesia corpus goes from 531 to 1056 MB/s for data compressed at level 1, and from 605 to 1222 MB/s for data compressed by zlib at level 6. Signed-off-by: Parker Loflin <parker@cronstable.dev>
encode_deflate_icf_aarch64 stores the bit buffer to memory after every symbol and loads it back, which puts store-to-load forwarding on the loop-carried dependency. Keep the bit buffer in registers and store it on exit. Combine the length code, distance code, and distance extra bits before shifting them into the bit buffer, so only one shift depends on the bit count. The compressed output does not change. On an Apple M4, deflate on the Silesia corpus goes from 381 to 460 MB/s at level 1 and from 348 to 409 MB/s at level 2. Signed-off-by: Parker Loflin <parker@cronstable.dev>
isal_deflate_icf_body_hash_hist_aarch64 writes each deflate_icf entry with three read-modify-write sequences, branches separately on the hash distance and on the match length, and compares matches 8 bytes at a time with each load address depending on the previous comparison. Rewrite the body. Write each entry with one 32-bit store. Compare the first 8 bytes of every candidate inline: when the hash distance is out of range, compare the input with itself and force the length to 0, so each position has one data-dependent branch. Compare longer matches 16 bytes per iteration and count the matching bytes once at the end. Compute the distance code without branches. The hash function, the hash table updates, and the compressed output do not change. On an Apple M4, deflate on the Silesia corpus goes from 460 to 578 MB/s at level 1 and from 408 to 585 MB/s at level 2. Signed-off-by: Parker Loflin <parker@cronstable.dev>
isal_deflate_body_aarch64 stores the bit buffer to the stream after every symbol, branches separately on the hash distance and on the match length, and compares matches with compare_258_bytes, whose next load address depends on the previous comparison. Rewrite the body like the level 1 and 2 body. Keep the bit buffer in registers and store it on exit. Compare the first 8 bytes of every candidate inline, with one data-dependent branch per position, and compare longer matches 16 bytes per iteration. Compute the distance code for distances above 2 without branches. Remove compare_258_bytes, which has no other users. The hash function, the hash table updates, and the compressed output do not change. On an Apple M4, level 0 deflate on the Silesia corpus goes from 575 to 681 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The aarch64 level 3 body, icf_body_hash1_fillgreedy_lazy(), calls gen_icf_map_h1_base() directly, so nothing uses gen_icf_map_h1_aarch64. That function also hashes with CRC32C instead of the compute_hash() that gen_icf_map_h1_base() and isal_deflate_hash_base() use, and on an Apple M4 it is no faster than the C function. Replace gen_icf_map_h1_aarch64 with a version that produces the same output as gen_icf_map_h1_base(), including the compute_hash() multiplicative hash. Its loop has no data-dependent branches: for each position it builds both the literal and the match entry and selects one. Select icf_body_lazyhash1_fillgreedy_greedy() for level 3, which reaches it through the gen_icf_map_lh1 dispatcher. The compressed output does not change. On an Apple M4, level 3 deflate on the Silesia corpus goes from 238 to 315 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
In the aarch64 inflate fast loop, the branch that separates literals from lengths mispredicts often, because whether a table entry ends in a length depends on the data. After the branch predicts a literal for an entry that ends in a length, the length path starts by loading the distance table entry. The branch itself resolves only after six instructions that depend on the table entry. Look up the distance table entry together with the next lit/len entry, before the branch: a distance code starts where the lit/len code ends. Test the last packed symbol with a shift and a mask instead of extracting it and comparing it with 256, so the branch resolves after four dependent instructions. On an Apple M4, isal_inflate_stateless() on the Silesia corpus goes from 1254 to 1383 MB/s for data compressed by zlib at level 6, and from 1060 to 1161 MB/s for data compressed by isa-l at level 1. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The aarch64 inflate fast loop runs only while at least 320 bytes of output space remain, enough for 2 literals, a 258-byte match, and the over-copy of the match. The per-symbol loop decodes the rest, which is the last 320 bytes of every stream and a large part of small streams. Within 320 bytes of the end of the output, check each symbol before decoding it. A literal entry writes at most 4 bytes, and a match of length len writes at most len + 33 bytes. Stay in the fast loop while the symbol fits and at least 64 bytes of space remain, and leave codes longer than ISAL_DECODE_LONG_BITS to the per-symbol loop. Both loops decode the same way, so the output and the return codes do not change. On an Apple M4, isal_inflate_stateless() of 1 KB, 4 KB, and 16 KB streams compressed at level 1 gets 5.8%, 3.1%, and 1.4% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The aarch64 level 0, 1, and 2 bodies count the matching bytes among the first 8 (eor, rbit, clz, and a shift) before the branch that separates literals from matches. That branch mispredicts often, and it resolves only after the count is known. Branch on a 4-byte comparison of the input and the candidate, folded into the distance range check with ccmp, and count the matching bytes only on the match path. A match needs 4 equal bytes, so the branch goes the same way as before. The compressed output does not change. On an Apple M4, deflate on the Silesia corpus goes from 693 to 739 MB/s at level 0, from 582 to 609 MB/s at level 1, and from 590 to 616 MB/s at level 2. Signed-off-by: Parker Loflin <parker@cronstable.dev>
gen_icf_map_h1_aarch64 handles one position per iteration with about 35 scalar instructions, and instruction throughput limits it. On an Apple M4, each instruction added to the loop slows level 3 by about 1.5%. Handle four positions per iteration. NEON computes the four compute_hash() values, the distance codes, and the deflate_icf entries. The hash table updates, the candidate loads, and the match lengths stay scalar and in order, so each position sees the hash table update of the position before it. The scalar loop handles the last positions. Its compute_hash() uses a 32-bit multiply for the second product, which gives the same masked hash because the hash mask is less than 1 << 16. The compressed output does not change. On an Apple M4, level 3 deflate on the Silesia corpus goes from 318 to 410 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
set_long_icf_fg_aarch64 checks one match map entry per iteration for an 8-byte match, extends those matches 8 bytes at a time, and copies a 128-byte distance table to the stack with memcpy() on every call. Find the entries with 8-byte matches 8 entries at a time with NEON, extend the matches 16 bytes at a time, and read the distance table in place. Remove the compare_aarch64 macro, which has no other users. The compressed output does not change. On an Apple M4, level 3 deflate on the Silesia corpus goes from 410 to 420 MB/s. Signed-off-by: Parker Loflin <parker@cronstable.dev>
build_huff_tree() repeatedly removes the two smallest keys from a binary heap and inserts the key of their parent. Each removal sifts a key down the heap with an unpredictable branch at each level. When level 1 compresses a 4 KB input on an Apple M4, build_heap() and build_huff_tree() take 28% of the time. Replace the C build_heap() and build_huff_tree() with a construction that gives the same tree. Sort the leaf keys with a radix sort on the frequency, and keep the keys of new nodes in the order they are made, which is ascending order of frequency. New nodes get decreasing ids, so among new nodes with the same frequency, the newest one has the smallest key. The smallest remaining key is then either the smallest leaf or the newest new node with the lowest frequency. The sort uses the tree nodes as scratch space, before they hold the tree. aarch64, ppc64le, riscv64, and base builds use the C functions, and x86 keeps its assembly versions. The compressed output does not change: for 900,000 random histograms, many of them with equal frequencies, the new functions build the same tree node for node. The exception is a histogram given to isal_create_hufftables() or isal_create_hufftables_subset() whose counts add up to 2^48 or more. The keys hold the counts in 48 bits, so the sums of keys wrap around, and the two constructions then give different code lengths, both of them valid. On an Apple M4, with the faster aarch64 deflate bodies of a separate patch series, stateless level 1 and 2 compression of 1 KB, 4 KB, 16 KB, and 64 KB inputs gets 35%, 23%, 8%, and 2% faster, and level 3 gets 29%, 16%, 4%, and 2% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The Huffman code generation for each deflate block has loops with data-dependent branches: init_heap32() and the other init_heap functions skip zero frequencies, fix_code_lens() handles leaves and internal nodes differently, and set_huff_codes() skips unused symbols. The init_heap functions also clear the whole 6.9 KB heap_tree for each of the three trees of a block, even for the 19 code length symbols. Store every key and advance past the nonzero ones, handle both kinds of tree node with the same stores, and assign codes to unused symbols too, which get the code 0 they already have. Clear only the tree nodes that build_huff_tree() fills. The compressed output does not change on aarch64 or x86. On an Apple M4, with the faster aarch64 deflate bodies of a separate patch series, stateless level 1 compression of 1 KB and 4 KB inputs gets 6% and 3% faster. On an AMD Ryzen 7 9800X3D, stateless level 1 and level 3 compression of 1 KB inputs gets 5% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
On x86, build_huff_tree() comes from proc_heap.asm, which keeps the keys in a binary heap. Each insertion and removal takes a data-dependent branch at every level of the heap. The assembly also writes only the child id of each tree node, so the init_heap functions clear the tree nodes before each tree. Use the C build_huff_tree() from proc_heap_base.c on every architecture. It gives the same tree and writes each node in full, so remove build_heap() and the clearing of the tree nodes, and remove proc_heap.asm and heap_macros.asm. The compressed output does not change, except, as on the other architectures, for histograms given to isal_create_hufftables() or isal_create_hufftables_subset() whose counts add up to 2^48 or more. On an AMD Ryzen 7 9800X3D, stateless level 1 compression of 1 KB, 4 KB, and 16 KB inputs gets 40%, 30%, and 12% faster, and level 3 compression 34%, 21%, and 8%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
Before it decodes a dynamic block, inflate decodes the code lengths in the block header and builds the lit/len and distance decode tables. On an AMD Ryzen 7 9800X3D this takes about 6,100 cycles for the header of a 1 KB block of Silesia, about half the time of inflating a 1 KB input, and 14,300 to 15,500 for a 64 KB block, depending on the compressor. The time goes to: - decoding the code lengths with the bit buffer in inflate_state, which every store to the code lengths can alias, and with a 10 bit table for the code length code, whose codes have at most 7 bits, - a branch on the code length of each of the 256 literals, and loads of counts and next codes right after they are stored, while giving the literals their codes and sorting them by length, - finding the long codes that share their first 12 bits by comparing every pair of long codes. Rewrite setup_dynamic_header() and the table builders: - Decode the code lengths into a byte array with the bit buffer in local variables and a 7 bit table, and add each literal to a list for its code length as it is decoded. - Give the literals of each length consecutive codes from those lists, then expand the length symbols, which leaves every expanded code sorted by length without a separate sort. - Group the long codes by their first 12 bits in one pass, numbering the groups in the short table entries that later point to them. - Build the code length code table by doubling it once per length, with the loop unrolled so the copies need no calls. The tables decode every input the same as before. The old code could count long codes of groups it had already handled as members of the group whose first 12 bits are all ones, which made that group's subtable bigger than its codes need. The new code sizes each subtable for its own codes, so the long code table can be laid out differently. For the headers of 3,035 dynamic blocks of Silesia slices compressed by zlib and by isa-l levels 1 and 3, 200,000 corrupted and truncated variants of them, and 200,000 headers with many long codes, the new code returns the same result, and its tables decode every input the same. Inflate gives the same output, return value, and remaining input as before for 40,000 valid and corrupted streams. The tables in static_inflate.h come out the same. On an AMD Ryzen 7 9800X3D, setting up the tables takes 38% fewer cycles for the headers of 1 KB blocks, 33% fewer for 4 KB and 16 KB blocks, and 37% fewer for 64 KB blocks. Inflate of 1 KB, 4 KB, 16 KB, and 64 KB slices of Silesia gets 21%, 10%, 5%, and 2% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
create_hufftables_icf() takes about 15,000 cycles for a 1 KB block on an AMD Ryzen 7 9800X3D, two thirds of the time of compressing a 1 KB input at level 1. It handles all 286 lit/len symbols at each step: it gathers them into a heap, writes a length and gives a code to each, adds up the length of the block over all of them, and run length encodes the code lengths with a branch on each one. fix_code_lens() reads 8 byte tree nodes whose depth was just stored as 4 bytes, which the CPU cannot forward from the store. Rewrite it around the symbols that occur: - make_code_lens() gathers them, builds the tree with build_huff_tree(), and visits it in the order fix_code_lens() does, with the depths in a byte array. It then limits the code lengths the same way. - set_codes_by_sym() gives codes to these symbols only, and the length of the block adds up only these symbols. - rl_encode_lens() finds each run of code lengths with 8 byte compares. - write_dynamic_header() writes each code length code together with its extra bits, with the bit buffer in local variables. isal_create_hufftables() and isal_create_hufftables_subset() now use rl_encode_lens() and write_dynamic_header() too, and the functions they replace are removed. sort_keys() sorts 32 keys or fewer by counting the smaller keys of each, which takes no branches, instead of with an insertion sort. The output does not change. create_hufftables_icf() gives the same tables, header bits, and return value for 500,000 random histograms, many with equal counts or counts that need their code lengths limited, and for the histograms of 2,454 blocks of Silesia. isal_create_hufftables() and isal_create_hufftables_subset() give the same tables for 300,000 random histograms, and deflate gives the same output for 40,000 random combinations of level, flush, wrapper, and buffer sizes. On an AMD Ryzen 7 9800X3D, create_hufftables_icf() takes 30%, 25%, 21%, and 18% fewer cycles for blocks of 1 KB, 4 KB, 16 KB, and 1 MB slices of Silesia. Level 1 and 2 compression of 1 KB, 4 KB, and 16 KB slices gets 25%, 12%, and 5% faster, and level 3 compression 22%, 8%, and 2%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
When inflate decodes the final block of a stream, it leaves pairs and triples of symbols out of the lit/len table when little input is left, because they take longer to add than they save: it builds single symbols for up to 2 KB of input, adds pairs for up to 4 KB, and adds triples above that. With the tables faster to build, single symbols are still best for 1 KB slices of Silesia, which compress to about 500 bytes, but on an AMD Ryzen 7 9800X3D pairs pay off from about 1 KB of input, and triples only from about 8 KB. Build single symbols for up to 1 KB of input, pairs for up to 8 KB, and triples above that. The decompressed data does not change, but inflate decodes pairs and triples as a whole. When the output buffer fills up, the input consumed can differ by a byte or two from before, and when the input is corrupted or ends too soon, inflate can stop one literal earlier. Continuing a valid stream gives the same result, and such differences could already happen between inputs of different sizes. On the same CPU, inflate of 3 KB slices of Silesia gets 1% to 3% faster, of 4 KB slices 2% to 3% faster, and of 8 KB slices about 1% faster, while inflate of 2 KB slices gets up to 2% slower. 1 KB slices and slices of 16 KB and more stay within about 1%. Single symbols for up to 1.5 KB of input lost most of the gain at 3 KB and did not measurably help at 2 KB. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The inflate decode tables keep codes longer than the short table index in long_code_lookup, which has 80 entries for the distance code and 1264 for the lit/len code. igzip_lib.h sizes it for the codes, entries that repeat codes shorter than the longest code of their group, and entries that no code uses, which only incomplete codes have. setup_dynamic_header() only rejects oversubscribed codes, so a header can define an incomplete code, and the bound on unused entries is too small: it allows 21 for the distance code, but a 15-bit code alone in the last group leaves 31 entries unused. An incomplete distance code can then need 86 entries: 30 codes, 25 repeats, and 31 unused entries. make_inflate_huff_code_dist() writes them without a check: it clears up to 12 bytes past long_code_lookup, which are the block_state, dict_length, and bfinal fields of inflate_state. master's make_inflate_huff_code_dist() has the same problem. Both table builders add up the space their long codes need before writing them, so return ISAL_INVALID_BLOCK if it is more than long_code_lookup holds. Complete codes always fit: a complete distance code needs at most 48 entries. zlib rejects incomplete codes as invalid, except a lit/len or distance code of a single code of length 1 and an empty distance code, none of which has long codes. The lit/len table gets the same check, since its size was worked out the same way, though no lit/len code was found that needs more than it holds. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The main loop of decode_huffman_code_block_stateless_01 and _04 looks up the next distance code with the bits of read_in after the refill, so the lookup waits for the refill. After the lit/len codes are consumed, read_in still holds at least 16 bits of the stream, enough for the 10-bit index of the distance lookup, so take the index before the refill, as the loop already does for the next lit/len lookup. On a Ryzen 7 9800X3D (Zen 5), inflate of the Silesia files gets 2% to 4% faster, and of 16 KB and 64 KB pieces of them 0.5% to 4% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
…hift A distance decode table entry holds the symbol, the number of its extra bits and the length of its code. The decoders shift read_in by the code length, then read the extra bits and shift read_in again, and only then can they look up the next lit/len code, so both shifts are on the dependency chain from one lit/len lookup to the next. Store the length of the code plus its extra bits in bits 4:0 of the entry instead, the symbol in bits 10:6 and the code length in bits 14:11. A decoder can then consume the code and the extra bits with one shift by the entry itself, since shrx and lsr use only the low 6 bits of the count, and read the extra bits off the critical path. An invalid entry still holds just the length of its code, and the entry that points to a subtable for longer codes moves its flag to bit 15, with the longest length in bits 10:7 and the offset in bits 6:0. Update the table builder and the C decoder, the x86 decoder and both loops of the aarch64 decoder, and regenerate static_inflate.h. On a Ryzen 7 9800X3D (Zen 5), inflate of isa-l level 1 streams of the Silesia files and of 64 KB pieces of them gets about 2% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
A lit/len decode table entry holds the length of its codes in bits 31:28, so the decoders shift it down before they can shift read_in, and the x86 decoder finds out whether the entry ends with a length or end of block symbol only after shifting out the last of its symbols. Both are on the paths that limit the main loop: the chain from one lit/len lookup to the next, and the branch between literals and a length, which is the branch that mispredicts most. Change the short_code_lookup entries so that the number of bits of their codes is in bits 4:0, and decoders can shift read_in by the entry itself. An entry for 1 to 3 literals holds their number in bits 7:6 and the literals from bit 8 on. An entry that ends with a length or end of block symbol has 0 in bits 7:6 and the symbol in bits 25:16, and if a literal comes first, the literal in bits 15:8 and LARGE_LIT_FLAG set. An invalid entry is 0, and an entry that points to a subtable for longer codes has LARGE_FLAG_BIT, now bit 5, set, the longest length in bits 4:0 and the offset from bit 8 on. As 8 bits are left for the third of three symbols, it must be a literal: where the tables had two literals and a length or end of block symbol, they now have the two literals, and the next lookup decodes the third symbol. In the x86 decoder, one test of the entry for a code longer than the lookup or an invalid code covers both, and one test of its symbol count tells literals from a length or end of block. The loop consumes the codes, writes the literals or the literal before a length, and does the lookups and the refill that both cases need before it branches on the symbol count. Update the table builder and the C decoder, the x86 decoder and both loops of the aarch64 decoder, and regenerate static_inflate.h. Without the triples that end with a length or end of block, inflate can stop at a different point when it runs out of output space or input: isal_inflate() can hold back fewer symbols in write_overflow_lits and leave more input unread, and on truncated input isal_inflate_stateless() can return up to two more literals of output. The output, and where the stream ends, are the same. On a Ryzen 7 9800X3D (Zen 5), inflate of isa-l level 1 streams of the Silesia files gets 12% to 13% faster, of 64 KB pieces of them about 11% faster, and of 4 KB pieces 5% to 6.5% faster. Signed-off-by: Parker Loflin <parker@cronstable.dev>
The main loop of the x86 inflate decoder stops OUT_BUFFER_SLOP bytes, 274, before the end of the output buffer, and leaves the rest to the loop at end_loop_block, which decodes a symbol at a time and copies matches with rep movsb. When the output buffer is the size of the output, as it often is when the caller knows it, that loop decodes the last 274 bytes, a quarter of a 1 KB stream. Within OUT_BUFFER_SLOP of the end, check whether the output of the next lookup fits before decoding it in the main loop: 8 bytes for literals, and for a length, the 1 byte of a literal before it plus the length plus COPY_SIZE, or plus 2 * COPY_SIZE for the stores of small_byte_copy. Only the last bytes, and codes longer than the lookup, go to end_loop_block. Also start the decoder functions on a 64-byte boundary, so that the main loop does not move with the code linked before it. On a Ryzen 7 9800X3D (Zen 5), inflate into an output buffer of the size of the output gets 13% to 16% faster for 1 KB inputs, 5% to 6% faster for 4 KB inputs and 1% to 3.5% faster for 16 KB inputs. Inflate of whole Silesia files changes by -1.4% to +0.6%, as it does with the 64-byte alignment alone. Signed-off-by: Parker Loflin <parker@cronstable.dev>
Author
|
Closing: this change is in #459 as its last two commits, "igzip: keep decoding near the end of the output in the x86 inflate loop" and "igzip: align the x86 inflate decoder functions to 64 bytes". |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR lets the main loop of the x86 inflate decoder keep decoding close to the end of the output buffer, which speeds up inflate into an output buffer of the size of the output, as callers that know the size use.
This PR is stacked on #459, and so on #440, #444 and #456. The commit for review is the last one.
Results
I measured on an AMD Ryzen 7 9800X3D (Zen 5) under WSL2. Each cell is
isal_inflate_stateless()throughput in MB/s, the median of three in-process A/B runs, each the best of 15 rounds that alternate the builds, into an output buffer of exactly the size of the output. "Whole files" inflates each Silesia file; the other columns inflate 256 pieces of the Silesia tar of that size, each compressed on its own.On the 9800X3D, with an output buffer larger than the output, inflate of 1 KB to 64 KB pieces is 0.7% to 4% faster than with #459, and of whole files 0.3% to 1.4% slower. The 64-byte alignment of the decoder functions accounts for the change on whole files: #459 with only that alignment added is 0.2% to 1.2% slower on whole files too. The gains above hold with #459 built at three other offsets within a cache line.
Other CPUs
I timed its base (
igzip-inflate-one-shift-tables) and this PR against each other on GitHub-hosted runners, with the workflow and harness in my fork (run, Intel and Zen 4 run, Zen 5, Ice Lake and Emerald Rapids run). The harness loads both builds'libisal.sointo one process and alternates them round by round. Each cell is its base (igzip-inflate-one-shift-tables) → this PR in MB/s, the median over the runs on that CPU (two per runner) of the best of 11 rounds, and in parentheses the median change of the per-round medians. Whole-file cases are the Silesia files, and slices are 256 pieces of the Silesia tar.Changes
The main loop of the x86 inflate decoder stops
OUT_BUFFER_SLOPbytes, 274, before the end of the output buffer, and leaves the rest to the loop atend_loop_block, which decodes a symbol at a time and copies matches withrep movsb. When the output buffer is the size of the output, that loop decodes the last 274 bytes, a quarter of a 1 KB stream.Within
OUT_BUFFER_SLOPof the end, the main loop checks whether the output of the next lookup fits before it decodes it: 8 bytes for literals, and for a length, the byte of a literal before it plus the length plusCOPY_SIZE, or plus2 * COPY_SIZEfor the stores ofsmall_byte_copy. Only the last bytes, and codes longer than the lookup, go toend_loop_block.The decoder functions also start on a 64-byte boundary, so that the main loop does not move with the code linked before it.
Testing
make -f Makefile.unx checkpasses, also with inflate dispatched todecode_huffman_code_block_stateless_01.PROT_NONEpage pass with each x86 decoder.ISAL_INVALID_LOOKBACK. For them, this PR reports 4 to 46 more bytes intotal_outand 8 bytes lessavail_in: it reaches the bad distance in the main loop, whose error exits on master count the bytes of the failed copy intotal_outand reportavail_inshort by the 8 bytes of input slop, which igzip: fix x86 inflate counts when the main loop finds an error #446 corrects.igzip_rand_testand keep all callee-saved registers.igzip_decode_block_stateless_01.asmand_04.asmassemble for elf64, win64 and macho64.