[ptweezy]

This PR fixes three igzip bugs, one per commit.

Reset the match history when a full flush is pending

Stateful FULL_FLUSH compression can read memory before the caller's input buffer and produce a valid deflate stream that inflates to the wrong data, at every level and on every architecture. This happens when the output buffer fills before the end of the block is written and the caller supplies more input before the flush completes.

When a call compresses all of its input into a compressed block under FULL_FLUSH, get_hist_size() drops the history, but the match history is reset only when sync_flush() writes the sync marker. The next call compresses its new input before writing the marker, with the old hash table but without the history it refers to, so match candidates point up to 32 KB before the new input buffer.

To reproduce, call isal_deflate() with FULL_FLUSH and 1 to 91 bytes of output space per call, add 1 to 117 bytes of input whenever avail_in reaches 0, and check the result with zlib. On the first 512 KB of dickens, master faults at levels 0 to 3 when each input chunk follows a PROT_NONE page. When each chunk follows unrelated text, master's output is wrong in 1 to 4 of 20 runs at level 0 and in all 20 at levels 1 to 3, although zlib accepts every stream as valid. With this fix, no run faults or gives wrong output.

If the call that leaves the flush pending compresses from the internal buffer instead, the history there (earlier NO_FLUSH or SYNC_FLUSH input, or a dictionary) stays. When the next call's input does not fit next to it, isal_deflate() compresses with flush set to NO_FLUSH, so sync_flush() writes the pending marker without resetting has_hist, and the input after the marker is compressed against the data before it. The stream inflates correctly, but igzip_lib.h documents that FULL_FLUSH makes the following blocks independent. A test that leaves the marker pending after each chunk's first call (where the output size allows it) and then supplies 1 byte to 64 KB of new input (dickens, 400 streams per level) finds data that refers back across 293 of master's 2,620 sync markers. It also hits the first case: 3 of master's level 3 streams inflate to the wrong data.

The fix sets has_hist to IGZIP_NO_HIST when a call compresses all input into a compressed block while a full flush is pending, so the next call resets the match history before compressing new input. In the same test, the data after each of the 2,666 sync markers decompresses on its own. Output is byte-identical to master's for callers that keep the flush type until each flush completes and supply more input only after it does, as the existing igzip_rand_test cases do.

The commit adds two tests to igzip_rand_test. test_full_flush_pending compresses at each level with FULL_FLUSH and small output buffers, and supplies more input while a flush is pending. test_full_flush_pending_marker leaves the sync marker pending at the start of a stream, after a SYNC_FLUSH and after a dictionary, supplies 40 KB more input, and checks that the data after the marker decompresses on its own.

Store the ICF buffer space in bytes

icf_buf_avail_out holds bytes everywhere except in update_state() (igzip_icf_base.c) and the aarch64 level 1 and 2 body and finish functions, which store an entry count. Each call that leaves the block open cuts the remaining space to a quarter, so blocks end early. Stateless level 1 and 2 compression is affected too: the finish function runs right after the body and sees a quarter of the space the body left, which with ISAL_DEF_LVL1_MIN or ISAL_DEF_LVL2_MIN level buffers is room for at most 217 ICF entries, so it can close the block early and put the rest of the input in another block. The x86 assembly already stores bytes, so x86 output does not change.

With the fix, stateful level 1 and 2 compression of dickens in 4 KB, 16 KB and 64 KB input buffers gives the same size as a single call. On master, aarch64 level 1 with 4 KB buffers gives 45.11% instead of 45.01%; arch=noarch is affected too.

Label igzip_perf inflate results by inflate mode

igzip_perf labels the stateless, stateful and zlib inflate runs with the deflate mode, so all three print the same name.

Conflicts

This PR conflicts with #448 in igzip/igzip_rand_test.c.

Testing

On x86-64 Linux (Ubuntu 24.04, GCC 13.3) and aarch64 Linux under qemu-user 8.2 (GCC 13.3):

• At every commit, tools/check_format.sh passes, and make -f Makefile.unx check passes on x86-64, with arch=noarch, and on aarch64.
• igzip_rand_test passes with seeds 1 to 20 on x86-64 and 1 to 10 on aarch64.
• Without the first commit, on x86-64, test_full_flush_pending fails with the default seed and 16 of seeds 1 to 20 (its buffer sizes come from rand(), so the failing seeds depend on the platform), and test_full_flush_pending_marker fails after the SYNC_FLUSH or the dictionary with the default seed and 19 of seeds 1 to 20.
• A differential test that supplies new input only after each flush completes (levels 0 to 3, stateless and stateful, all flush types, gzip and zlib wrappers, hist_bits, dictionaries, random buffer sizes, every input buffer ending at a PROT_NONE page) matches master in all 30,000 x86-64 configurations. On aarch64, all 3,773 configurations at level 0 and 3,766 at level 3 match; at levels 1 and 2, the second fix changes 976 of 3,762 and 1,057 of 3,699 streams. zlib inflates all 7,775 finished raw deflate, gzip and zlib streams without a preset dictionary.