Repository navigation
Conversation
After a match of 4 to 7 bytes, the next position depends on the match length, so the search runs as a chain: the length, a load at the new position, its hashes, the hash table entries and the candidates. When such a match starts at the first of the two positions checked per iteration, the bytes after it lie in the 8 bytes that start 4 bytes into the match. Load those before the match is known and hash the two positions after the match from them, shifted by the match length. A 4 to 7 byte match after a literal still loads the bytes after it once its length is known. Compute minus the distance of a candidate as (table[hash] - pos) | ~dist_mask, two instructions instead of four, and read the input with scalar loads instead of keeping 16 bytes in xmm4. The body's fixed-size loads reach at most 11 bytes past the current position, within the 16 bytes of input padding. The output does not change. Level 1 and 2 in-process A/B on the Silesia corpus, Ryzen 7 9800X3D, against master: whole files +8% to +9%, 4 KB slices +3% to +4%, 16 KB slices +6%, 64 KB slices +7% to +8%. Signed-off-by: Parker Loflin <parker@cronstable.dev>
ptweezy
force-pushed
the
igzip-x86-short-match-hash
branch
from
October 21, 2026 21:27
a07afc7 to
ae6e9b1
Compare
ptweezy
marked this pull request as draft
October 21, 2026 22:01
ptweezy
marked this pull request as ready for review
October 21, 2026 22:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After a match of 4 to 7 bytes, the x86 level 1 and 2 match search cannot hash the next position until it knows the length and has loaded the bytes there. When the match starts at the first of the two positions each iteration checks, those bytes lie within the 8 bytes starting 4 bytes into the match. This PR loads those 8 bytes before the length is known and hashes the next two positions from them, shifted by the length. It also computes a candidate's negated distance as
(table[hash] - pos) | ~dist_mask(two instructions instead of four) and reads the input with scalar loads instead of keeping 16 bytes in xmm4.CPUs and builds
Every x86-64 level 1 and 2 body in
igzip_icf_body_h1_gr_bt.asmchanges:_01(SSE4.2),_02(AVX) and_04(AVX2 and AVX-512 CPUs). Levels 0 and 3 do not use this code. Output does not change.Results
master → this PR in MB/s on GitHub-hosted runners, level 1 unless noted: in-process A/B on the Silesia files and 256 slices of the Silesia tar, median over all runs (two per runner) of the best of 11 rounds. In parentheses, the median over runs of the change in median-round throughput (harness; runs 1, 2, 3).
Testing
make checkpasses with the level 1 and 2 dispatch forced to each of_01,_02and_04, and natively on Windows (mingw-w64, Ryzen 7 9800X3D), where a win64 thunk also finds xmm6-xmm15 and the callee-saved GPRs intact across every body call. The file also assembles for macho64.avail_in,avail_outand output on every call (stateless and stateful, sync and stateless full flush, all wrappers,hist_bits, dictionaries, guard-page inputs). Silesia sizes at levels 1 and 2 are identical.