diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b14d0b63f..c7e44a33d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -348,9 +348,10 @@ jobs: - name: FFI parity + cross-validation tests # The libzstd-linking parity / cross-validation / corpus tests live in # `ffi-bench`; `bench-internals` activates the white-box-facade targets - # gated behind it. Corpus fixtures resolve from the manifest dir, so no + # gated behind it, and `dict-builder` the trainer parity targets. + # Corpus fixtures resolve from the manifest dir, so no # working-directory is needed. - run: cargo nextest run --profile ci -p ffi-bench --features bench-internals + run: cargo nextest run --profile ci -p ffi-bench --features bench-internals,dict-builder cross-i686: needs: lint diff --git a/README.md b/README.md index ac4c9e734..c9407b1b2 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ - **Production-grade decoder** — complete [RFC 8878](https://www.rfc-editor.org/rfc/rfc8878) implementation: dictionary-backed streams, raw / RLE / compressed blocks, the full frame format, optional content checksums, runtime-dispatched SIMD kernels (SSE2 / BMI2 / AVX2 / NEON, opt-in AVX-512). - **Full-range encoder** — every C-zstd level (`-131072..=22`) produces valid frames decodable by this crate and by upstream C zstd; named presets, per-knob parameter overrides, long-distance matching, streaming via `std::io::Write`. -- **Dictionaries end to end** — compress and decompress with the same dictionary format C zstd consumes; reusable parsed handles; pure-Rust COVER / FastCOVER training behind the `dict-builder` feature. +- **Dictionaries end to end** — compress and decompress with the same dictionary format C zstd consumes; reusable parsed handles; pure-Rust COVER / FastCOVER / legacy training behind the `dict-builder` feature. - **Wire-compatible both ways** — frames interoperate with C zstd in either direction; interop is enforced in CI against the reference implementation. - **`no_std` ready** — the decoder builds with `--no-default-features` for embedded and sandboxed targets. - **WebAssembly / npm** — the same codec as an npm package with automatic SIMD selection; no native addons, no postinstall scripts. @@ -88,9 +88,15 @@ sound archive. `--exclude-compressed` skips inputs whose extension names an already-compressed format. -Flags that only steer how the work is done (`-T`, `-B`, `--adapt`, ...) are -accepted and ignored; their values are still validated, so a typo is an error -rather than silence. +Flags that only steer how the work is done (`-T`, `--adapt`, `-B` when +compressing, ...) are accepted and ignored; their values are still validated, +so a typo is an error rather than silence. `-b` measures every input as frames +of its own, cut into `-B#` pieces when a size is given, as upstream's +benchmark does. +`--max` sets every compression parameter to its hardest end, as upstream's +does, with the window stopped at 27 (the widest this build decodes), and +`--show-default-cparams` prints what the level selects for each input in +upstream's layout. `--target-compressed-block-size` does take effect: it bounds what goes into a block, so blocks flush sooner. `--long` means `--long=27`, as upstream documents, and is capped there: a larger window would produce frames this @@ -105,16 +111,20 @@ anything but zstd and `--rsyncable`, which needs the worker threads this build does not have. `-M` is treated as the safety promise it is: on the runs that decode, a limit covering the 128 MiB window, the decoder's buffers and the `-D` dictionary is kept and a tighter one is refused rather than -ignored. Compressing, listing and training allocate no decoder, so the flag is -accepted there and describes nothing, as upstream has it. +ignored. Compressing and listing allocate no decoder, so the flag is accepted +there and describes nothing, as upstream has it; for `--train-legacy` it caps +the samples loaded, as below. `--train` and `--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#]` train with FastCOVER, the algorithm upstream also defaults to (a knob set to zero keeps its default, as upstream reads it), and a bare `--train-cover` trains with the COVER trainer. Its tuning, `--train-cover=...`, is refused rather than misread: the reference-side parameters name knobs this trainer does not -have. `--train-legacy` names an algorithm this build does not have and is -refused. `-D` takes either a dictionary produced by `--train` or any file at +have. `--train-legacy[=s=#]` (or `-s#`) runs upstream's original trainer, which +counts samples: they are loaded as upstream loads them (each file one sample of +up to 128 KiB, or cut into `-B#` pieces, whole samples up to 2 GiB or `-M` when +that is smaller), and for the same file list the dictionary carries the same +content as upstream's. `-D` takes either a dictionary produced by `--train` or any file at all, which is then used as raw content the way upstream does; such a dictionary has no ID, so the same bytes must be supplied when decoding. @@ -248,6 +258,9 @@ in pure Rust: - COVER (`create_raw_dict_from_source`) and FastCOVER (`create_fastcover_raw_dict_from_source`) raw dictionaries - `finalize_raw_dict` to produce the full zstd dictionary format - `create_fastcover_dict_from_source` for train + finalize in one call +- `create_legacy_dict_from_slice`: upstream's original suffix-array trainer + (`ZDICT_trainFromBuffer_legacy`), whose content matches upstream's byte for + byte on the same samples ## Feature flags @@ -261,7 +274,7 @@ in pure Rust: | `kernel-simd128` | ✅ | WebAssembly SIMD kernel (needs `-C target-feature=+simd128`) | | `kernel-vbmi2` | ❌ | AVX-512 decode kernel (see note below) | | `kernel-scalar` | ✅ | Marker for the always-compiled scalar fallback | -| `dict-builder` | ❌ | Pure-Rust COVER / FastCOVER dictionary training | +| `dict-builder` | ❌ | Pure-Rust COVER / FastCOVER / legacy dictionary training | | `lsm` | ❌ | [Storage-format extensions](#storage-format-extensions) | Each flag gates its tier wherever that tier exists. `kernel-sse`, diff --git a/ffi-bench/Cargo.toml b/ffi-bench/Cargo.toml index 760683e44..fa4f22a4a 100644 --- a/ffi-bench/Cargo.toml +++ b/ffi-bench/Cargo.toml @@ -133,7 +133,12 @@ required-features = ["bench-internals"] [[test]] name = "dictionary_ffi" path = "tests/dictionary_ffi.rs" -required-features = ["bench-internals"] +required-features = ["bench-internals", "dict-builder"] + +[[test]] +name = "legacy_trainer_ffi" +path = "tests/legacy_trainer_ffi.rs" +required-features = ["bench-internals", "dict-builder"] [[test]] name = "encode_corpus_ffi" diff --git a/ffi-bench/tests/legacy_trainer_ffi.rs b/ffi-bench/tests/legacy_trainer_ffi.rs new file mode 100644 index 000000000..1d9af99a2 --- /dev/null +++ b/ffi-bench/tests/legacy_trainer_ffi.rs @@ -0,0 +1,134 @@ +//! The legacy trainer against the reference's `ZDICT_trainFromBuffer_legacy`: +//! both walk the same suffix array with the same selection rules, so the +//! content they choose has to be the same bytes. Only the header in front of it +//! differs, since the entropy tables are built by each side's own finalizer. +#![cfg(all(feature = "bench-internals", feature = "dict-builder"))] + +use structured_zstd::testing::legacy_dict_content; +use zstd::zstd_safe::zstd_sys; + +/// Every sample back to back, and the length of each. +type Samples = (Vec, Vec); + +/// The reference's dictionary content for the same corpus: its dictionary with +/// the header (magic, id, entropy tables, repeat offsets) cut off. +fn reference_content( + samples: &[u8], + sizes: &[usize], + dict_size: usize, + selectivity: u32, +) -> Option> { + let mut dict = vec![0u8; dict_size]; + let params = zstd_sys::ZDICT_legacy_params_t { + selectivityLevel: selectivity, + zParams: zstd_sys::ZDICT_params_t { + compressionLevel: 0, + notificationLevel: 0, + dictID: 0, + }, + }; + // SAFETY: every buffer is valid for the length passed with it, and + // `sizes` holds `sizes.len()` entries summing to `samples.len()`. + let written = unsafe { + zstd_sys::ZDICT_trainFromBuffer_legacy( + dict.as_mut_ptr().cast(), + dict.len(), + samples.as_ptr().cast(), + sizes.as_ptr(), + sizes.len() as u32, + params, + ) + }; + // SAFETY: plain query on a return code. + if written == 0 || unsafe { zstd_sys::ZDICT_isError(written) } != 0 { + return None; + } + dict.truncate(written); + // SAFETY: `dict` holds `written` bytes of the dictionary just built. + let header = unsafe { zstd_sys::ZDICT_getDictHeaderSize(dict.as_ptr().cast(), dict.len()) }; + // SAFETY: plain query on a return code. + assert_eq!(unsafe { zstd_sys::ZDICT_isError(header) }, 0); + Some(dict[header..].to_vec()) +} + +/// Log lines of a few shapes, one sample each. +fn log_lines(count: u32) -> Samples { + const SHAPES: [&str; 4] = [ + "ts={i} level=INFO msg=\"flush memtable\" tenant=demo table=orders region=eu-west\n", + "ts={i} level=WARN msg=\"slow compaction\" tenant=demo table=users region=us-east\n", + "ts={i} level=INFO msg=\"rotate segment\" tenant=acme table=orders region=eu-west\n", + "ts={i} level=ERROR msg=\"write stalled\" tenant=acme table=events region=ap-south\n", + ]; + let mut samples = Vec::new(); + let mut sizes = Vec::new(); + for i in 0..count { + let line = SHAPES[(i % 4) as usize].replace("{i}", &format!("{:08}", i * 7919)); + sizes.push(line.len()); + samples.extend_from_slice(line.as_bytes()); + } + (samples, sizes) +} + +/// The systemd unit files under `dict_tests/files`, one sample each. +fn unit_files() -> Samples { + let dir = concat!(env!("CARGO_MANIFEST_DIR"), "/../zstd/dict_tests/files"); + let mut names: Vec<_> = std::fs::read_dir(dir) + .expect("the fixture directory exists") + .map(|entry| entry.expect("readable entry").path()) + .filter(|path| path.extension().is_none_or(|ext| ext != "zst")) + .collect(); + names.sort(); + let mut samples = Vec::new(); + let mut sizes = Vec::new(); + for name in names { + let bytes = std::fs::read(&name).expect("readable fixture"); + if bytes.is_empty() { + continue; + } + sizes.push(bytes.len()); + samples.extend_from_slice(&bytes); + } + (samples, sizes) +} + +/// The decodecorpus file cut into 4 KiB samples, as `--train -B4096` cuts it. +fn corpus_blocks() -> Samples { + let bytes = include_bytes!("../../zstd/decodecorpus_files/z000033").to_vec(); + let sizes = bytes.chunks(4096).map(<[u8]>::len).collect(); + (bytes, sizes) +} + +/// Same corpus, same selectivity, same size: the same content, byte for byte. +#[test] +fn the_legacy_trainer_selects_the_references_content() { + let fixtures: [(&str, Samples); 3] = [ + ("log lines", log_lines(3000)), + ("unit files", unit_files()), + ("decodecorpus blocks", corpus_blocks()), + ]; + for (name, (samples, sizes)) in &fixtures { + for selectivity in [0u32, 4, 9, 12] { + for dict_size in [4096usize, 16 * 1024, 112_640] { + let ours = legacy_dict_content(samples, sizes, dict_size, selectivity); + let theirs = reference_content(samples, sizes, dict_size, selectivity); + let (Some(ours), Some(theirs)) = (ours, theirs) else { + panic!("{name} s={selectivity} size={dict_size}: one side trained nothing"); + }; + // The reference places its header in front of the content and + // lets it overwrite the front when both do not fit, so its + // content is ours or a tail of it. + assert!( + theirs.len() <= ours.len() && ours.ends_with(&theirs), + "{name} s={selectivity} size={dict_size}: {} content bytes against the \ + reference's {}", + ours.len(), + theirs.len(), + ); + assert!( + theirs.len() * 10 >= ours.len() * 9, + "{name} s={selectivity} size={dict_size}: the header should cost a sliver" + ); + } + } + } +} diff --git a/zstd/src/bin/structured-zstd/main.rs b/zstd/src/bin/structured-zstd/main.rs index 3fe8d6d0b..fd221d493 100644 --- a/zstd/src/bin/structured-zstd/main.rs +++ b/zstd/src/bin/structured-zstd/main.rs @@ -15,8 +15,8 @@ use std::io::{self, BufReader, ErrorKind, IsTerminal, Read, Write}; use std::path::{Path, PathBuf}; use structured_zstd::encoding::{ - CompressionContext, CompressionLevel, CompressionParameters, LiteralCompressionMode, Strategy, - StreamingEncoder, + CompressionContext, CompressionLevel, CompressionParameters, LevelParameters, + LiteralCompressionMode, Strategy, StreamingEncoder, }; /// Error type for the tool: a boxed message, which is all a command-line @@ -142,6 +142,10 @@ struct Options { /// Measure each input on its own (`-S`) instead of as one stream, so the /// reported ratio and throughput describe a file rather than a mixture. bench_separately: bool, + /// `-B#`: the benchmark cuts every input into independent frames of this + /// many bytes (below [`MIN_BENCH_BLOCK_SIZE`] it cuts nothing, as the + /// reference's benchmark ignores such a size). + block_size: Option, /// Long-distance matching (`--long`), enabled on the encoder via the /// compression-parameters API. long: bool, @@ -212,10 +216,16 @@ struct Options { /// with the window sized to the input, so the whole reference is /// reachable. patch_from: Option, + /// Print the parameters the level selects for each input before + /// compressing it (`--show-default-cparams`). + show_default_cparams: bool, /// Which dictionary trainer `--train*` runs. trainer: Trainer, /// The trainer's tuning from `--train-fastcover=...` / `--train-cover=...`. trainer_params: TrainerParams, + /// The legacy trainer's selectivity from `-s#` or `--train-legacy=s=#`; + /// zero is its default. + selectivity: u32, } /// The dictionary trainers `--train` selects between. @@ -225,6 +235,8 @@ enum Trainer { FastCover, /// `--train-cover`: the segment-scoring COVER trainer. Cover, + /// `--train-legacy`: the reference's original suffix-array trainer. + Legacy, } /// Tuning from `--train-fastcover=k=#,d=#,f=#,steps=#,split=#,accel=#` and @@ -282,6 +294,11 @@ const DEFAULT_MAX_DICT: usize = 112_640; /// Least time `-b` measures each level for (upstream `BMK_TIMETEST_DEFAULT_S`). const DEFAULT_BENCH_SECONDS: f64 = 3.0; +/// Smallest `-B` block the benchmark cuts its inputs into; a smaller one leaves +/// each input whole (`benchzstd.c`, `BMK_benchMemAdvancedNoAlloc`: +/// `adv->blockSize >= 32`). +const MIN_BENCH_BLOCK_SIZE: u64 = 32; + /// Window log a bare `--long` selects, as upstream documents (128 MiB). const DEFAULT_LONG_WINDOW_LOG: u32 = 27; @@ -454,7 +471,7 @@ fn check_window_log(log: u32) -> Result<()> { let bounds = CParameter::WindowLog.bounds(); let decodable = structured_zstd::decoding::MAXIMUM_ALLOWED_WINDOW_SIZE.ilog2(); - let upper = bounds.upper_bound.min(i64::from(decodable)); + let upper = i64::from(max_window_log()); if i64::from(log) < bounds.lower_bound || i64::from(log) > upper { bail!( "window log {log} is outside the supported range {}..={upper} \ @@ -466,6 +483,50 @@ fn check_window_log(log: u32) -> Result<()> { Ok(()) } +/// The largest window this build both writes and reads back: the encoder's +/// ceiling or the decoder's, whichever is lower (see [`check_window_log`]). +fn max_window_log() -> u32 { + use structured_zstd::encoding::CParameter; + + let encodable = u32::try_from(CParameter::WindowLog.bounds().upper_bound) + .expect("the window-log bound is a small positive number"); + let decodable = structured_zstd::decoding::MAXIMUM_ALLOWED_WINDOW_SIZE.ilog2(); + encodable.min(decodable) +} + +/// Every knob at the end of its range that compresses hardest, as the +/// reference command's `--max` sets them (`zstdcli.c`, `setMaxCompression`). +/// One departure: the window stops at [`max_window_log`] rather than at 31, +/// since a larger one would write frames this build refuses to decode. The +/// long-distance hash rate is left to derive from the rest, which is what the +/// reference's 0 there asks for. +fn max_compression_params() -> AdvancedParams { + use structured_zstd::encoding::CParameter; + + let upper = |parameter: CParameter| { + u32::try_from(parameter.bounds().upper_bound) + .expect("every compression-parameter bound is a small positive number") + }; + let lower = |parameter: CParameter| { + u32::try_from(parameter.bounds().lower_bound) + .expect("every compression-parameter bound is a small positive number") + }; + AdvancedParams { + window_log: Some(max_window_log()), + chain_log: Some(upper(CParameter::ChainLog)), + hash_log: Some(upper(CParameter::HashLog)), + search_log: Some(upper(CParameter::SearchLog)), + min_match: Some(lower(CParameter::MinMatch)), + target_length: Some(upper(CParameter::TargetLength)), + strategy: Some(Strategy::Btultra2), + ldm_hash_log: Some(upper(CParameter::LdmHashLog)), + // The reference's heuristic value, not a bound. + ldm_min_match: Some(16), + ldm_bucket_size_log: Some(upper(CParameter::LdmBucketSizeLog)), + ldm_hash_rate_log: None, + } +} + /// Validate the parameter list of `--adapt=min=N,max=N`. /// /// The bounds have no effect here — the level does not vary — but a command @@ -898,6 +959,7 @@ fn parse_args_into( bench_end: default_level, bench_secs: DEFAULT_BENCH_SECONDS, bench_separately: false, + block_size: None, long: false, long_window_log: None, memory_limit: None, @@ -926,8 +988,10 @@ fn parse_args_into( advanced: AdvancedParams::default(), literals: LiteralCompressionMode::Auto, patch_from: None, + show_default_cparams: false, trainer: Trainer::FastCover, trainer_params: TrainerParams::default(), + selectivity: 0, }; let mut ultra = false; // `-e`, when typed. Without it the benchmark ends where it starts, so no @@ -975,11 +1039,9 @@ fn parse_args_into( opts.trainer = Trainer::Cover; opts.trainer_params = TrainerParams::default(); } - // The legacy trainer produces a different dictionary. Accepting - // the flag and running another trainer would hand back one the - // caller did not ask for, with nothing to say so. "train-legacy" => { - bail!("--{long} is not implemented; --train-cover and --train-fastcover are") + select_mode(&mut opts, Mode::Train); + opts.trainer = Trainer::Legacy; } // `-c` and `-o` name competing destinations, so each clears the // other and the later one on the command line wins, as upstream @@ -999,6 +1061,18 @@ fn parse_args_into( "keep" => opts.keep = true, "rm" => opts.remove_source = true, "ultra" => ultra = true, + // Replaces every knob at once, as the reference command's does, + // so a `--zstd=` before it is overwritten and one after it + // adjusts the maximum. Its tables at their widest do not fit a + // 32-bit address space, which the reference refuses the same way. + "max" => { + if usize::BITS < 64 { + bail!("--max is incompatible with 32-bit mode"); + } + ultra = true; + opts.long = true; + opts.advanced = max_compression_params(); + } "quiet" => *verbosity -= 1, "verbose" => *verbosity += 1, // The wire-format switches: the checksum, the @@ -1014,6 +1088,7 @@ fn parse_args_into( "no-pass-through" => opts.pass_through = Some(false), "exclude-compressed" => opts.exclude_compressed = true, "ignore-read-errors" => opts.ignore_read_errors = true, + "show-default-cparams" => opts.show_default_cparams = true, "progress" => opts.progress = Progress::Always, "no-progress" => opts.progress = Progress::Never, "version" => { @@ -1116,9 +1191,12 @@ fn parse_args_into( } else if let Some(v) = option_text(long, "threads", arg_os, &mut iter)? { let _ = v.parse::().wrap_err("invalid --threads")?; } else if let Some(v) = option_text(long, "block-size", arg_os, &mut iter)? { - // The job size of a multi-threaded run: nothing here, - // but a malformed size is still a broken command line. - parse_size(&v).wrap_err("invalid --block-size")?; + // The long spelling of `-B#`: the reference reads both + // into one setting (zstdcli.c, `--block-size` and `-B`), + // which cuts the benchmark's frames and the training + // samples, and is a multi-threaded job size otherwise. + let size = parse_size(&v).wrap_err("invalid --block-size")?; + opts.block_size = (size != 0).then_some(size); } else if let Some(list) = option_value(long, "filelist", arg_os, &mut iter)? { opts.filelists.push(list); } else if let Some(dir) = @@ -1156,6 +1234,23 @@ fn parse_args_into( select_mode(&mut opts, Mode::Train); opts.trainer = Trainer::FastCover; opts.trainer_params = parse_trainer_params(v, true)?; + } else if let Some(v) = long.strip_prefix("train-legacy=") { + // `s=#` or `selectivity=#`, as `parseLegacyParameters` + // reads it. + select_mode(&mut opts, Mode::Train); + opts.trainer = Trainer::Legacy; + let value = v + .strip_prefix("selectivity=") + .or_else(|| v.strip_prefix("s=")) + .ok_or_else(|| { + eyre!("--train-legacy takes `s=#` or `selectivity=#`, got `{v}`") + })?; + let (selectivity, tail) = read_leading_u32(value) + .wrap_err("invalid --train-legacy selectivity")?; + if !tail.is_empty() { + bail!("invalid --train-legacy selectivity `{value}`"); + } + opts.selectivity = selectivity; } else if let Some(reference) = option_value(long, "patch-from", arg_os, &mut iter)? { @@ -1247,27 +1342,47 @@ fn parse_args_into( 'k' => opts.keep = true, // `-S` measures each input on its own. 'S' => opts.bench_separately = true, + 's' => { + // `-s#`: the legacy trainer's selectivity, read like the + // reference reads it (`readU32FromChar`). + let rest: String = chars[ci + 1..].iter().collect(); + let (selectivity, tail) = + read_leading_u32(&rest).wrap_err("invalid -s selectivity")?; + if !tail.is_empty() { + bail!("invalid -s selectivity `{rest}`"); + } + opts.selectivity = selectivity; + ci = chars.len(); + continue; + } 'q' => *verbosity -= 1, 'v' => *verbosity += 1, 'C' => opts.checksum = true, 'r' => opts.recursive = true, - 'B' | 'T' => { - // `-B[N]` job / block size, `-T[N]` thread count. Both - // steer how the work is done, not what comes out: we use a - // fixed block size and run single-threaded. Upstream - // accepts them, so a script that passes them must not fail - // here — but the VALUE is still parsed: ignoring what a - // flag does is not a reason to ignore what it says, and a - // typo is a broken command line either way. A size takes a - // size suffix; a thread count is a plain count, the way - // `--threads=` reads it. + 'B' => { + // `-B[N]` cuts the benchmark's inputs into independent + // frames and a training sample into several samples. When + // compressing it is the job size of a multi-threaded run, + // which this build does not have, so it is kept and has no + // effect there. Read as the reference reads it + // (`readU32FromChar`): a count with an optional `K` / `M`. + let rest: String = chars[ci + 1..].iter().collect(); + let (size, tail) = read_leading_u32(&rest).wrap_err("invalid -B value")?; + if !tail.is_empty() { + bail!("invalid -B value `{rest}`"); + } + opts.block_size = (size != 0).then_some(u64::from(size)); + ci = chars.len(); + continue; + } + 'T' => { + // `-T[N]` thread count: single-threaded here, so it steers + // nothing, but the value is still parsed the way + // `--threads=` reads it, since a typo is a broken command + // line either way. let rest: String = chars[ci + 1..].iter().collect(); if !rest.is_empty() { - if c == 'B' { - parse_size(&rest).wrap_err("invalid -B value")?; - } else { - rest.parse::().wrap_err("invalid -T thread count")?; - } + rest.parse::().wrap_err("invalid -T thread count")?; } ci = chars.len(); continue; @@ -1609,6 +1724,8 @@ Advanced options: Advanced compression options: --ultra Enable levels beyond 19, up to 22; requires more memory. + --max Compress with every parameter at its maximum; the window stops at 27, + the widest this build reads back. Requires a lot of memory. --fast[=#] Use to very fast compression levels. [Default: 1] --long[=#] Enable long distance matching with window log #. [Default: 27] Available from level 16 up (or with --zstd=strat=7..9), where @@ -1618,6 +1735,7 @@ Advanced compression options: --zstd=wlog=#,clog=#,hlog=#,slog=#,mml=#,tlen=#,strat=#[,lhlog=#,lmml=#,lblog=#,lhrlog=#] Override the level's compression parameters knob by knob. --exclude-compressed Only compress files that are not already compressed. + --show-default-cparams Print the parameters the level selects for each input. --stream-size=# Specify size of streaming input from STDIN. --size-hint=# Optimize compression parameters for streaming input of approximately size #. @@ -1641,6 +1759,10 @@ Dictionary builder: --train-cover Use the cover algorithm (takes no tuning here). --train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#] Use the fast cover algorithm (with optional arguments). + + --train-legacy[=s=#] Use the legacy algorithm with selectivity #. [Default: 9] + -B# With --train-legacy, cut each file into samples of size #; + otherwise each file is one sample of up to 128 KiB. -o NAME Use NAME as dictionary name. [Default: dictionary] --maxdict=# Limit dictionary to specified size #. [Default: 112640] --dictID=# Force dictionary ID to #. [Default: Random] @@ -1649,22 +1771,106 @@ Benchmark options: -b# Perform benchmarking with compression level #. [Default: 3] -e# Test all compression levels up to #; starting level is `-b#`. [Default: 1] -i# Set the minimum evaluation to time # seconds. [Default: 3] + -B# Cut file into independent chunks of size #. [Default: No chunking] -S Output one benchmark result per input file. [Default: Consolidated result] -D dictionary Benchmark using dictionary Environment: ZSTD_CLEVEL sets the default compression level; ZSTD_NBTHREADS is read and validated. Accepted for compatibility, with no effect here: -T#/--threads=#, --single-thread, ---auto-threads, -B#, --block-size=#, --adapt, --zstd=ovlog=#, --[no-]sparse, +--auto-threads, -B# and --block-size=# when compressing (the job size of a +multi-threaded run), --adapt, --zstd=ovlog=#, --[no-]sparse, --[no-]asyncio, --[no-]mmap-dict, --[no-]row-match-finder (compression runs single-threaded). Rejected rather than ignored, because they would change the result: --format= -other than zstd, --rsyncable (needs worker threads), --train-legacy, shrink in -the trainer tuning, and -M/--memory below the enforced ceiling when decoding. +other than zstd, --rsyncable (needs worker threads), shrink in the trainer +tuning, and -M/--memory below the enforced ceiling when decoding. +--train-cover and --train-fastcover read whole files, so -B does not cut them. A new output file keeps its source's permissions. "; +/// Upstream's names for the nine strategies, in ordinal order from 1 +/// (`zstdcli.c`, `ZSTD_strategyMap`). +const STRATEGY_NAMES: [&str; 9] = [ + "ZSTD_fast", + "ZSTD_dfast", + "ZSTD_greedy", + "ZSTD_lazy", + "ZSTD_lazy2", + "ZSTD_btlazy2", + "ZSTD_btopt", + "ZSTD_btultra", + "ZSTD_btultra2", +]; + +/// Write what `--show-default-cparams` reports for one input: the parameters +/// `level` selects for it, in the reference command's layout +/// (`zstdcli.c`, `printDefaultCParams`). +/// +/// `size` is the input's length when it has one (`None` for stdin or anything +/// not a regular file). A length of zero is printed as such but sized as an +/// unknown source, because `ZSTD_getCParams`, which the reference calls here, +/// reads zero as "unknown". +fn write_default_cparams( + out: &mut impl Write, + name: &str, + size: Option, + dictionary_size: usize, + level: i32, +) -> io::Result<()> { + match size { + Some(bytes) => writeln!(out, "{name} ({bytes} bytes)")?, + None => writeln!(out, "{name} (src size unknown)")?, + } + let params = + LevelParameters::for_level(level, size.filter(|&bytes| bytes != 0), dictionary_size); + let ordinal = params.strategy.ordinal(); + // `ordinal` is 1..=9 by construction of `Strategy`. + let strategy = STRATEGY_NAMES[ordinal as usize - 1]; + writeln!(out, " - windowLog : {}", params.window_log)?; + writeln!(out, " - chainLog : {}", params.chain_log)?; + writeln!(out, " - hashLog : {}", params.hash_log)?; + writeln!(out, " - searchLog : {}", params.search_log)?; + writeln!(out, " - minMatch : {}", params.min_match)?; + writeln!(out, " - targetLength : {}", params.target_length)?; + writeln!(out, " - strategy : {strategy} ({ordinal})") +} + +/// Print `--show-default-cparams` for every input of a compressing run, on +/// stderr and whatever the verbosity, as the reference command does. +fn show_default_cparams(opts: &Options) -> Result<()> { + let dictionary_size = match dictionary_path(opts) { + Some(path) => fs::metadata(path) + .wrap_err_with(|| format!("failed to inspect dictionary file {}", path.display()))? + .len(), + None => 0, + }; + let dictionary_size = usize::try_from(dictionary_size) + .map_err(|_| eyre!("dictionary of {dictionary_size} bytes does not fit in memory"))?; + let mut err = io::stderr().lock(); + let stdin = [PathBuf::from("-")]; + let inputs: &[PathBuf] = if opts.inputs.is_empty() { + &stdin + } else { + &opts.inputs + }; + for input in inputs { + let (name, size) = if input == Path::new("-") { + (STDIN_MARK.to_string(), None) + } else { + let size = fs::metadata(input) + .ok() + .filter(fs::Metadata::is_file) + .map(|metadata| metadata.len()); + (input.display().to_string(), size) + }; + write_default_cparams(&mut err, &name, size, dictionary_size, opts.level) + .wrap_err("failed to write the default parameters")?; + } + Ok(()) +} + /// The file the run's dictionary comes from: `-D`, or the `--patch-from` /// reference, which is a dictionary by another name. The command line refuses /// both at once, so at most one is set. @@ -2047,6 +2253,20 @@ fn run_selected(mut opts: Options) -> Result { return list_files(&opts); } + // Reached only by the streaming modes, as the reference command's check is + // (benchmark, training and listing have left by now and ignore the flag). + // Decompression has no parameters to show; testing is not decompression + // there, so it is not refused, and prints nothing since it compresses + // nothing. + if opts.show_default_cparams { + if opts.mode == Mode::Decompress { + bail!("error : can't use --show-default-cparams in decompression mode"); + } + if opts.mode == Mode::Compress { + show_default_cparams(&opts)?; + } + } + // A destination named outright belongs to the whole run, whatever it reads: // stdin, an explicit `-`, or files. The dictionary is what the frame being // written will need to be read back, so an `-o` pointing at it destroys the @@ -2575,17 +2795,27 @@ fn run_benchmark(opts: &Options, dict: Option>) -> Result<()> { if let Some(limit) = opts.memory_limit { // With `-S` only one input is in memory at a time, so the largest file // is what has to fit rather than their sum. + let subject: &[u64] = if opts.bench_separately { + let at = sizes + .iter() + .position(|&size| size == largest) + .expect("the largest size is one of the sizes"); + &sizes[at..=at] + } else { + &sizes + }; let inputs = if opts.bench_separately { largest } else { sum }; - // Three buffers exist at once: the input, the frame it compresses to, + // Three buffers exist at once: the input, the frames it compresses to, // and the decoded copy. Each is allocated at the size named here and // never grows past it, so this is what the run actually holds rather - // than a lower bound on it. The frame's is `compress_bound`, which is - // the input plus the framing an incompressible input still pays — the - // case a ceiling has to survive. - let frame = usize::try_from(inputs) - .map(structured_zstd::encoding::compress_bound) - .map(|bound| bound as u64) - .ok(); + // than a lower bound on it. The frames' is `compress_bound` of each, + // which is the input plus the framing an incompressible input still + // pays — the case a ceiling has to survive. The encoder is sized by the + // frame it builds, so the widest frame is the source it is weighed + // against. + let extent = bench_frames_extent(subject, opts.block_size); + let frame = extent.map(|(room, _)| room); + let widest_chunk = extent.map_or(0, |(_, widest)| widest); // Beside them stands the match finder every compression pass builds, // whose tables are the largest thing at the higher levels — hundreds of // MiB where the buffers are tens. It is sized by the level, by the @@ -2621,13 +2851,13 @@ fn run_benchmark(opts: &Options, dict: Option>) -> Result<()> { Some(parameters) => { structured_zstd::encoding::estimated_compression_workspace_bytes_for_parameters( ¶meters, - Some(inputs), + Some(widest_chunk), dictionary, ) } None => structured_zstd::encoding::estimated_compression_workspace_bytes_for_run( compression_level, - Some(inputs), + Some(widest_chunk), None, false, dictionary, @@ -2670,7 +2900,13 @@ fn run_benchmark(opts: &Options, dict: Option>) -> Result<()> { for (input, size) in opts.inputs.iter().zip(&sizes) { let data = read_inputs_bounded(std::slice::from_ref(input), std::slice::from_ref(size))?; - benchmark_one(opts, codecs, &input.display().to_string(), &data)?; + benchmark_one( + opts, + codecs, + &input.display().to_string(), + &data, + std::slice::from_ref(size), + )?; } return Ok(()); } @@ -2682,7 +2918,7 @@ fn run_benchmark(opts: &Options, dict: Option>) -> Result<()> { [only] => only.display().to_string(), many => format!(" {} files", many.len()), }; - benchmark_one(opts, codecs, &label, &data) + benchmark_one(opts, codecs, &label, &data, &sizes) } /// Read every input into one buffer, taking no more room — and no more bytes — @@ -2719,9 +2955,12 @@ fn read_inputs_bounded(inputs: &[PathBuf], sizes: &[u64]) -> Result> { file.take(size + 1) .read_to_end(&mut data) .wrap_err_with(|| format!("failed to read {}", input.display()))?; - if (data.len() - before) as u64 > *size { + // Short is refused as well as long: the frames are cut at the sizes + // recorded before the read, so a file that shrank would be cut past + // the end of what was read. + if (data.len() - before) as u64 != *size { bail!( - "{} grew while it was being read; run again", + "{} changed size while it was being read; run again", input.display() ); } @@ -2729,10 +2968,67 @@ fn read_inputs_bounded(inputs: &[PathBuf], sizes: &[u64]) -> Result> { Ok(data) } -/// Measure one benchmark subject: the whole input as one stream, or a single -/// file under `-S`. Split out so the two modes differ only in what they hand -/// over, not in how the measurement is taken. -fn benchmark_one(opts: &Options, codecs: &mut Codecs, label: &str, data: &[u8]) -> Result<()> { +/// The lengths of the independent frames a benchmark compresses its inputs as, +/// in order: every input a frame of its own, cut into `block_size` pieces when +/// one of at least [`MIN_BENCH_BLOCK_SIZE`] is given. An empty input yields no +/// frame. This is the reference's block table (`benchzstd.c`, +/// `BMK_benchMemAdvancedNoAlloc`), so ratio and speed describe the same frames. +/// +/// Yielded, not listed: with a small `-B` over a large input a list would hold +/// a word per frame that the `-M` ceiling never counted. +fn bench_chunk_lengths( + file_sizes: &[u64], + block_size: Option, +) -> impl Iterator + '_ { + let block = block_size.filter(|&size| size >= MIN_BENCH_BLOCK_SIZE); + file_sizes.iter().flat_map(move |&size| { + let piece = block.unwrap_or(size.max(1)); + (0..size.div_ceil(piece)).map(move |at| piece.min(size - at * piece)) + }) +} + +/// The frames [`bench_chunk_lengths`] cuts `file_sizes` into, measured without +/// listing them: the room they take at most (`compress_bound` of each) and the +/// widest of them. Worked out per file from its whole blocks and its tail, so a +/// small `-B` over a large input costs nothing before `-M` has weighed it. +/// `None` when the room is more than this machine can address. +fn bench_frames_extent(file_sizes: &[u64], block_size: Option) -> Option<(u64, u64)> { + let block = block_size.filter(|&size| size >= MIN_BENCH_BLOCK_SIZE); + let bound = |len: u64| -> Option { + u64::try_from(structured_zstd::encoding::compress_bound( + usize::try_from(len).ok()?, + )) + .ok() + }; + let mut room = 0u64; + let mut widest = 0u64; + for &size in file_sizes { + if size == 0 { + continue; + } + let piece = block.unwrap_or(size); + let whole = size / piece; + let tail = size % piece; + room = room.checked_add(whole.checked_mul(bound(piece)?)?)?; + if tail > 0 { + room = room.checked_add(bound(tail)?)?; + } + widest = widest.max(size.min(piece)); + } + Some((room, widest)) +} + +/// Measure one benchmark subject: every input together, or a single file under +/// `-S`. Split out so the two modes differ only in what they hand over, not in +/// how the measurement is taken. `file_sizes` are the lengths of the inputs +/// `data` holds, in order; each is compressed as frames of its own. +fn benchmark_one( + opts: &Options, + codecs: &mut Codecs, + label: &str, + data: &[u8], + file_sizes: &[u64], +) -> Result<()> { use std::time::Instant; if data.is_empty() { @@ -2750,42 +3046,65 @@ fn benchmark_one(opts: &Options, codecs: &mut Codecs, label: &str, data: &[u8]) ); if opts.verbosity == 1 { // The reference command's machine-readable header, for scripts that - // drive `-b -q`. + // drive `-b -q`; the block size is the one asked for, as it prints it. println!( - "bench {UPSTREAM_VERSION} : input {} bytes, {} seconds, 0 KB blocks", + "bench {UPSTREAM_VERSION} : input {} bytes, {} seconds, {} KB blocks", data.len(), - opts.bench_secs as u64 + opts.bench_secs as u64, + opts.block_size.unwrap_or(0) >> 10 ); } + debug_assert_eq!( + bench_chunk_lengths(file_sizes, opts.block_size).sum::(), + data.len() as u64, + "the frames cover the input exactly" + ); // The two buffers the measurement fills, sized once from what they will - // hold: the frame can be no larger than `compress_bound` says, and the - // decoded copy is exactly the input again. That keeps them the size the `-M` - // ceiling counted them at instead of the doubled capacity a growing `Vec` - // ends up with — and it keeps the growth out of the timed sections, which - // would otherwise be reported as compression and decompression speed. - let mut compressed = Vec::with_capacity(structured_zstd::encoding::compress_bound(data.len())); + // hold: the frames can be no larger than `compress_bound` of each says, + // and the decoded copy is exactly the input again. That keeps them the size + // the `-M` ceiling counted them at instead of the doubled capacity a + // growing `Vec` ends up with — and it keeps the growth out of the timed + // sections, which would otherwise be reported as compression and + // decompression speed. + let frames_bound = bench_frames_extent(file_sizes, opts.block_size) + .and_then(|(room, _)| usize::try_from(room).ok()) + .ok_or_else(|| eyre!("-b: {label} is more than this machine can hold compressed"))?; + let mut compressed = Vec::with_capacity(frames_bound); let mut decoded = Vec::with_capacity(data.len()); for level in opts.bench_start..=opts.bench_end { validate_level(level)?; + let settings = FrameSettings { + level, + size_hint: None, + // The reference's benchmark compresses with the library's frame + // defaults (`BMK_initCCtx` sets no checksum), not the command's, so + // its sizes and decoding speeds carry no content checksum. + checksum: false, + ..FrameSettings::from_options(opts) + }; let mut best_compress = f64::MAX; let start = Instant::now(); loop { compressed.clear(); let t = Instant::now(); - compress_stream( - data, - &mut compressed, - &FrameSettings { - level, - // The benchmark holds the whole input, so the length is - // exact and there is no estimate to fall back on. - pledged_size: Some(data.len() as u64), - size_hint: None, - ..FrameSettings::from_options(opts) - }, - codecs, - )?; + let mut rest = data; + for chunk in bench_chunk_lengths(file_sizes, opts.block_size) { + // Each piece is a frame of its own, as the reference's + // benchmark compresses every block independently. The length is + // exact, so it is pledged rather than estimated. + let (piece, tail) = rest.split_at(chunk as usize); + rest = tail; + compress_stream( + piece, + &mut compressed, + &FrameSettings { + pledged_size: Some(chunk), + ..settings + }, + codecs, + )?; + } best_compress = best_compress.min(t.elapsed().as_secs_f64()); if start.elapsed().as_secs_f64() >= opts.bench_secs { break; @@ -2808,6 +3127,11 @@ fn benchmark_one(opts: &Options, codecs: &mut Codecs, label: &str, data: &[u8]) break; } } + // A speed measured on output that is not the input measures nothing; + // the reference's benchmark checks the round trip the same way. + if decoded != data { + bail!("-b: level {level} did not decode {label} back to its input"); + } let c_speed = if best_compress > 0.0 { mb / best_compress @@ -2899,13 +3223,14 @@ fn bench_display_name(label: &str) -> String { } } -/// `--train`: build a FastCOVER dictionary from the concatenated sample files -/// and write it to `-o` (default `dictionary`). Mirrors upstream -/// `zstd --train FILEs -o dict --maxdict=N [--dictID=N]`. +/// `--train`: build a dictionary from the sample files with the selected +/// trainer (FastCOVER, COVER or legacy) and write it to `-o` (default +/// `dictionary`). Mirrors upstream `zstd --train FILEs -o dict --maxdict=N +/// [--dictID=N]`. fn train_dictionary(opts: &Options) -> Result<()> { use structured_zstd::dictionary::{ - FinalizeOptions, create_fastcover_dict_from_slice, create_raw_dict_from_slice, - finalize_raw_dict, + FinalizeOptions, create_fastcover_dict_from_slice, create_legacy_dict_from_slice, + create_raw_dict_from_slice, finalize_raw_dict, }; if opts.inputs.iter().any(|input| input == Path::new("-")) { @@ -2942,9 +3267,17 @@ fn train_dictionary(opts: &Options) -> Result<()> { // Whether the trainer takes the tuning it was given is a question about the // command line alone, so it is answered before any sample is touched: a run // bound to be refused does not first read a corpus that may be large. - // `Some` holds the FastCOVER options, `None` stands for COVER. - let fastcover = match opts.trainer { - Trainer::FastCover => Some(fastcover_options(&opts.trainer_params)?), + enum Plan { + FastCover(structured_zstd::dictionary::FastCoverOptions), + Cover, + Legacy, + } + let plan = match opts.trainer { + Trainer::FastCover => Plan::FastCover(fastcover_options(&opts.trainer_params)?), + // The legacy trainer is tuned by selectivity alone; a cover tuning list + // given before `--train-legacy` names a trainer that no longer runs, + // as it does in the reference. + Trainer::Legacy => Plan::Legacy, Trainer::Cover => { // The COVER trainer here scores segments by k-mer frequency, as the // reference's does, but is not parameterised the same way: `k`, @@ -2957,7 +3290,7 @@ fn train_dictionary(opts: &Options) -> Result<()> { use --train-fastcover=... for a tunable trainer" ); } - None + Plan::Cover } }; let output = opts @@ -3009,37 +3342,33 @@ fn train_dictionary(opts: &Options) -> Result<()> { } } - // Each sample is opened once, and what the dictionary may carry is taken - // from that same open file rather than from its path afterwards. A path - // answers about whatever it names at the moment it is asked, and training - // takes long enough for a sample to be replaced while it runs: asking again - // at the end could describe a file whose bytes are not the ones now inside - // the dictionary, and grant its permissions to theirs. - let mut corpus = Vec::new(); - let mut samples = Vec::with_capacity(opts.inputs.len()); - for input in &opts.inputs { - let mut file = File::open(input) - .wrap_err_with(|| format!("failed to open training sample {}", input.display()))?; - let metadata = file - .metadata() - .wrap_err_with(|| format!("failed to inspect {}", input.display()))?; - if !metadata.is_file() { - bail!( - "--train needs regular files: {} is not one", - input.display() - ); - } - file.read_to_end(&mut corpus) - .wrap_err_with(|| format!("failed to read training sample {}", input.display()))?; - samples.push(metadata); - } - let finalize = FinalizeOptions { dict_id: opts.dict_id, }; let mut dict = Vec::new(); - match fastcover { - Some(options) => { + let sources = match plan { + Plan::Legacy => { + // The legacy trainer counts samples, so they are loaded as the + // reference's command loads them: shuffled, capped per file, and + // cut by `-B`. The same files then yield the same content. + // `-M` caps what is loaded, as the reference's command passes its + // memory limit to `DiB_trainFromFiles` (zstdcli.c), which keeps + // whole samples up to it (dibio.c); dropping it would train on a + // different corpus than the reference for the same command line. + let set = load_training_samples(&opts.inputs, opts.block_size, opts.memory_limit)?; + create_legacy_dict_from_slice( + &set.corpus, + &set.sizes, + &mut dict, + opts.max_dict, + opts.selectivity, + finalize, + ) + .map_err(|err| eyre!("dictionary training failed: {err}"))?; + set.sources + } + Plan::FastCover(options) => { + let (corpus, sources) = read_whole_samples(&opts.inputs)?; // From the slice, not through a reader: the corpus is the largest // thing this run holds, and the reader path buffers it a second // time inside. @@ -3051,8 +3380,10 @@ fn train_dictionary(opts: &Options) -> Result<()> { finalize, ) .map_err(|err| eyre!("dictionary training failed: {err}"))?; + sources } - None => { + Plan::Cover => { + let (corpus, sources) = read_whole_samples(&opts.inputs)?; // From the slice, as FastCOVER is: the reader path would buffer // the whole corpus a second time, and `corpus` has to stay alive // for the finalizing pass below anyway. @@ -3067,8 +3398,9 @@ fn train_dictionary(opts: &Options) -> Result<()> { // was asked for, so the best content survives the cut. dict = finalize_raw_dict(raw.as_slice(), corpus.as_slice(), opts.max_dict, finalize) .map_err(|err| eyre!("dictionary training failed: {err}"))?; + sources } - } + }; // A trained dictionary is an output file like any other, so it is written // through a temporary that is renamed into place: an interrupted run @@ -3081,7 +3413,7 @@ fn train_dictionary(opts: &Options) -> Result<()> { .and_then(|()| sink.flush()) .wrap_err_with(|| format!("failed to write dictionary {}", output.display()))?; } else { - place_trained_dictionary(&output, &dict, &samples)?; + place_trained_dictionary(&output, &dict, &sources)?; } display!( opts.verbosity, @@ -3094,6 +3426,206 @@ fn train_dictionary(opts: &Options) -> Result<()> { Ok(()) } +/// Read every sample file whole into one corpus, for the trainers that do not +/// tell samples apart; `-B` has nothing to cut for them, and they get every +/// byte, as the reference's do when `-B` cuts files into samples. +/// +/// Each sample is opened once, and what the dictionary may carry is taken from +/// that same open file rather than from its path afterwards. A path answers +/// about whatever it names at the moment it is asked, and training takes long +/// enough for a sample to be replaced while it runs: asking again at the end +/// could describe a file whose bytes are not the ones now inside the +/// dictionary, and grant its permissions to theirs. +fn read_whole_samples(inputs: &[PathBuf]) -> Result<(Vec, Vec)> { + let mut corpus = Vec::new(); + let mut sources = Vec::with_capacity(inputs.len()); + for input in inputs { + let mut file = File::open(input) + .wrap_err_with(|| format!("failed to open training sample {}", input.display()))?; + let metadata = file + .metadata() + .wrap_err_with(|| format!("failed to inspect {}", input.display()))?; + if !metadata.is_file() { + bail!( + "--train needs regular files: {} is not one", + input.display() + ); + } + file.read_to_end(&mut corpus) + .wrap_err_with(|| format!("failed to read training sample {}", input.display()))?; + sources.push(metadata); + } + Ok((corpus, sources)) +} + +/// Most bytes one file contributes as a sample when `-B` does not cut it +/// (`dibio.c`, `SAMPLESIZE_MAX`). +const TRAINING_SAMPLE_MAX: u64 = 128 << 10; + +/// Most training data loaded at all (`dibio.c`, `MAX_SAMPLES_SIZE`). +const TRAINING_DATA_MAX: u64 = 2 << 30; + +/// Fewest samples a trainer is given (`dibio.c`: "nb of samples too low"). +const TRAINING_SAMPLES_MIN: usize = 5; + +/// The samples a trainer is handed, loaded the way the reference's command +/// loads them (`dibio.c`, `DiB_trainFromFiles`). +struct TrainingSet { + /// Every sample back to back. + corpus: Vec, + /// The length of each sample in `corpus`. + sizes: Vec, + /// What each file read was, taken from the open file. + sources: Vec, +} + +/// Reorder the sample files the way the reference does before loading +/// (`DiB_shuffle`), so a sample set too large to load keeps a spread of files +/// rather than the first ones, and the corpus is laid out as there. +/// +/// The same loop step for step (dibio.c, `DiB_shuffle` and `DiB_rand`, alike in +/// 1.5.7 and later): from the last position down to 1, swap with +/// `rand % (i + 1)`, the generator seeded with `0xFD2FB528`. +fn shuffle_training_files(files: &mut [T]) { + let mut seed: u32 = 0xFD2F_B528; + let mut next = || { + seed = (seed.wrapping_mul(2_654_435_761) ^ 2_246_822_519).rotate_left(13); + seed >> 5 + }; + for i in (1..files.len()).rev() { + let j = (next() % (i as u32 + 1)) as usize; + files.swap(i, j); + } +} + +/// How many samples `file_sizes` make and how many bytes they hold, as the +/// loader would take them without a limit: each file one sample of at most +/// [`TRAINING_SAMPLE_MAX`] bytes, or cut into `block_size` samples; empty files +/// left out. +fn training_extent(file_sizes: &[u64], block_size: Option) -> (u128, u128) { + // Summed exactly: sparse files report any length, and the sum of a slice of + // `u64`s cannot outgrow `u128` (a slice holds fewer than 2^64 of them). + let mut wanted = 0u128; + let mut samples = 0u128; + for &size in file_sizes { + if size == 0 { + continue; + } + let (count, bytes) = match block_size { + Some(block) => (size.div_ceil(block), size), + None => (1, size.min(TRAINING_SAMPLE_MAX)), + }; + samples += u128::from(count); + wanted += u128::from(bytes); + } + (samples, wanted) +} + +/// Load the training samples as the reference's command does: files in its +/// shuffled order, each one sample of at most [`TRAINING_SAMPLE_MAX`] bytes, or +/// cut whole into `block_size` samples when `-B` gives one; empty files left +/// out; at most [`TRAINING_DATA_MAX`] bytes, or `memory_limit` when smaller. +fn load_training_samples( + inputs: &[PathBuf], + block_size: Option, + memory_limit: Option, +) -> Result { + let mut order: Vec<&PathBuf> = inputs.iter().collect(); + shuffle_training_files(&mut order); + + let mut file_sizes = Vec::with_capacity(order.len()); + for input in &order { + file_sizes.push( + fs::metadata(input) + .wrap_err_with(|| format!("failed to inspect {}", input.display()))? + .len(), + ); + } + let (samples, wanted) = training_extent(&file_sizes, block_size); + if samples < TRAINING_SAMPLES_MIN as u128 { + bail!( + "{samples} training sample(s) is too few; provide one file per sample, or \ + split files into fixed-size samples with -B#" + ); + } + // The budget holds each sample's recorded length as well as its bytes: under + // a small `-B` the lengths outweigh the bytes they describe, a word per byte + // at `-B1`, and a limit on the bytes alone would be exceeded by the list. + const SIZE_ENTRY: u64 = core::mem::size_of::() as u64; + let budget = TRAINING_DATA_MAX.min(memory_limit.unwrap_or(u64::MAX)); + // The two buffers share the budget, so their room is split from it rather + // than each given all of it. A sample holds at most `piece` bytes and costs + // an entry, so the bytes are at most `piece / (piece + entry)` of the + // budget; the entries are one per whole piece that fits, plus one per file + // for the shorter piece at its end. + let piece = block_size.unwrap_or(TRAINING_SAMPLE_MAX); + let files = file_sizes.iter().filter(|&&size| size > 0).count() as u64; + // In `u128`: `piece` is whatever `-B` asked for, up to `u64::MAX`. + let per_sample = u128::from(piece) + u128::from(SIZE_ENTRY); + let bytes_room = u128::from(budget) * u128::from(piece) / per_sample; + let corpus_room = usize::try_from(wanted.min(bytes_room)) + .map_err(|_| eyre!("{budget} bytes of samples is more than this machine can hold"))?; + let entries = samples.min(u128::from(budget) / per_sample + u128::from(files)); + let entries = usize::try_from(entries) + .map_err(|_| eyre!("{entries} samples is more than this machine can hold"))?; + + let mut set = TrainingSet { + corpus: Vec::with_capacity(corpus_room), + sizes: Vec::with_capacity(entries), + sources: Vec::new(), + }; + 'files: for input in order { + if set.sizes.len() as u128 >= samples { + break; + } + let file = File::open(input) + .wrap_err_with(|| format!("failed to open training sample {}", input.display()))?; + let metadata = file + .metadata() + .wrap_err_with(|| format!("failed to inspect {}", input.display()))?; + let size = metadata.len(); + if size == 0 { + continue; + } + let mut reader = file; + let mut taken = 0u64; + loop { + let piece = match block_size { + Some(block) => (size - taken).min(block), + None => size.min(TRAINING_SAMPLE_MAX), + }; + // What is held with this sample's entry, never more than a word past + // the budget; compared without adding `piece`, which a large `-B` + // makes as large as the file. + let held = set.corpus.len() as u64 + (set.sizes.len() as u64 + 1) * SIZE_ENTRY; + if held > budget || piece > budget - held { + if taken == 0 { + break 'files; + } + break; + } + let before = set.corpus.len(); + (&mut reader) + .take(piece) + .read_to_end(&mut set.corpus) + .wrap_err_with(|| format!("failed to read training sample {}", input.display()))?; + if (set.corpus.len() - before) as u64 != piece { + bail!( + "{} changed while it was being read; run again", + input.display() + ); + } + set.sizes.push(piece as usize); + taken += piece; + if block_size.is_none() || taken >= size || set.sizes.len() as u128 >= samples { + break; + } + } + set.sources.push(metadata); + } + Ok(set) +} + /// Write a trained dictionary to the regular file `output` through a /// temporary renamed into place, no more readable than the strictest of the /// `samples` (their metadata) it was trained on. diff --git a/zstd/src/bin/structured-zstd/tests.rs b/zstd/src/bin/structured-zstd/tests.rs index 64463e147..f0b93ba85 100644 --- a/zstd/src/bin/structured-zstd/tests.rs +++ b/zstd/src/bin/structured-zstd/tests.rs @@ -943,6 +943,111 @@ fn the_benchmark_input_is_the_size_it_was_counted_at() { assert_eq!(combined[8000], 2, "the second file's follow"); } +/// The `-M` check measures the benchmark's frames without listing them; what it +/// works out has to be what the list itself adds up to, for whole blocks, +/// tails, empty inputs and block sizes below the minimum alike. +#[test] +fn the_frame_extent_matches_the_frames_it_describes() { + let files: [&[u64]; 4] = [&[], &[0, 1, 5000], &[1 << 20, 3], &[70_000, 0, 131_071]]; + for sizes in files { + for block in [None, Some(1024), Some(MIN_BENCH_BLOCK_SIZE), Some(40_000)] { + let chunks: Vec = bench_chunk_lengths(sizes, block).collect(); + let room: u64 = chunks + .iter() + .map(|&len| structured_zstd::encoding::compress_bound(len as usize) as u64) + .sum(); + let widest = chunks.iter().copied().max().unwrap_or(0); + assert_eq!( + bench_frames_extent(sizes, block), + Some((room, widest)), + "{sizes:?} in blocks of {block:?}" + ); + } + } +} + +/// The training budget holds the sample sizes as well as the samples. Under +/// `-B1` every byte is a sample whose recorded length is a word, eight times +/// the byte it describes, so a budget spent on the bytes alone was exceeded +/// several times over by the list beside them. +#[test] +fn training_sample_sizes_count_against_the_budget() { + let input = std::env::temp_dir().join(format!("szstd-train-b1-{}", std::process::id())); + fs::write(&input, [3u8; 64]).unwrap(); + + let set = load_training_samples(std::slice::from_ref(&input), Some(1), Some(200)); + + let _ = fs::remove_file(&input); + let set = set.expect("the samples load within the budget"); + assert!(!set.sizes.is_empty(), "some samples fit"); + // Reserved as well as used: both buffers are sized from the one budget, + // which allows a word per file for the partial sample at its end. + let entry = core::mem::size_of::(); + assert!( + set.corpus.capacity() + set.sizes.capacity() * entry <= 200 + entry, + "{} bytes of room for samples and {} for sizes exceed the 200-byte budget", + set.corpus.capacity(), + set.sizes.capacity() + ); + assert!( + set.corpus.len() + set.sizes.len() * core::mem::size_of::() <= 200, + "{} bytes of samples and {} sizes exceed the 200-byte budget", + set.corpus.len(), + set.sizes.len() + ); +} + +/// A block size at the top of the range makes each file one sample, and the +/// budget arithmetic that divides by a sample's cost, or compares a piece with +/// what is left, must not overflow on it. +#[test] +fn a_block_size_at_the_top_of_the_range_loads_whole_files() { + let input = std::env::temp_dir().join(format!("szstd-train-bmax-{}", std::process::id())); + fs::write(&input, [5u8; 700]).unwrap(); + + let set = load_training_samples( + &[ + input.clone(), + input.clone(), + input.clone(), + input.clone(), + input.clone(), + ], + Some(u64::MAX), + None, + ); + + let _ = fs::remove_file(&input); + let set = set.expect("five files are five samples"); + assert_eq!(set.sizes, vec![700; 5]); +} + +/// File sizes that add up past `u64::MAX` (sparse files report any length) are +/// summed exactly rather than wrapped or stopped at a bound: both the byte +/// total and the sample count are the true ones. +#[test] +fn training_sizes_past_the_integer_range_do_not_wrap() { + let huge = u64::MAX / 2 + 1; + let (samples, wanted) = training_extent(&[huge, huge, 0], Some(4096)); + assert_eq!(wanted, 2 * u128::from(huge)); + assert_eq!(samples, 2 * u128::from(huge.div_ceil(4096))); +} + +/// A benchmark input that shrank between being sized and being read is +/// refused. The frames are cut at the sizes recorded first, so a short read +/// would cut past the end of the buffer instead of measuring anything. +#[test] +fn a_benchmark_input_that_shrank_is_refused() { + let input = std::env::temp_dir().join(format!("szstd-benchshrink-{}", std::process::id())); + fs::write(&input, vec![7u8; 1000]).unwrap(); + + let read = read_inputs_bounded(std::slice::from_ref(&input), &[4000]); + + let _ = fs::remove_file(&input); + let err = read.expect_err("a file shorter than its recorded size is an error"); + assert!(err.to_string().contains("changed"), "{err}"); +} + /// Permission bits alone do not say who they let in. Two samples at `0640` may /// belong to different groups, and the dictionary belongs to whichever group the /// directory it was created in gave it — so keeping the group bits would open @@ -1724,19 +1829,119 @@ fn training_refuses_to_overwrite_without_force() { } /// The trainer flags name algorithms, and the algorithm decides what the -/// dictionary contains. FastCOVER and COVER are here; the legacy trainer is -/// not, so its flag has to say no rather than run another trainer under its -/// name. +/// dictionary contains: `--train-legacy` selects the legacy trainer, and its +/// selectivity comes from `=s=#`, `=selectivity=#` or `-s#`, read as the +/// reference reads them. #[test] -fn the_legacy_trainer_is_refused_not_substituted() { +fn the_legacy_trainer_flags_select_it_and_its_selectivity() { assert_eq!(parse(&["--train", "s1"]).unwrap().mode, Mode::Train); assert_eq!( parse(&["--train-fastcover", "s1"]).unwrap().mode, Mode::Train ); assert_eq!(parse(&["--train-cover", "s1"]).unwrap().mode, Mode::Train); - assert!(parse(&["--train-legacy", "s1"]).is_err()); - assert!(parse(&["--train-legacy=s=8", "s1"]).is_err()); + let legacy = parse(&["--train-legacy", "s1"]).unwrap(); + assert_eq!( + (legacy.mode, legacy.trainer), + (Mode::Train, Trainer::Legacy) + ); + assert_eq!(legacy.selectivity, 0, "the trainer's own default"); + assert_eq!(parse(&["--train-legacy=s=8", "s1"]).unwrap().selectivity, 8); + assert_eq!( + parse(&["--train-legacy=selectivity=12", "s1"]) + .unwrap() + .selectivity, + 12 + ); + assert_eq!( + parse(&["--train-legacy", "-s5", "s1"]).unwrap().selectivity, + 5 + ); + assert!(parse(&["--train-legacy=k=8", "s1"]).is_err(), "no such key"); + assert!( + parse(&["--train-legacy=s=8x", "s1"]).is_err(), + "trailing junk" + ); + assert!(parse(&["--train-legacy", "-sx", "s1"]).is_err()); +} + +/// `--train-legacy` trains end to end: sample files in, a dictionary out that +/// the decoder accepts, made of what the samples repeat. +#[test] +fn legacy_training_writes_a_dictionary() { + let dir = Scratch::new("legacy-train"); + let mut names = Vec::new(); + for i in 0..40u32 { + let body = format!( + "[Unit]\nDescription=worker {i}\nAfter=network-online.target\n\n[Service]\n\ + ExecStart=/usr/bin/worker --id {i} --config /etc/worker/worker.toml\n\ + Restart=on-failure\n" + ); + names.push(dir.file(&format!("w{i}.service"), body.as_bytes())); + } + let output = dir.path().join("legacy.dict"); + let mut args = vec!["--train-legacy".to_string(), "-o".to_string()]; + args.push(output.display().to_string()); + args.extend(names.iter().map(|name| name.display().to_string())); + let argv: Vec<&str> = args.iter().map(String::as_str).collect(); + run(parse(&argv).unwrap()).unwrap(); + + let dict = fs::read(&output).unwrap(); + structured_zstd::decoding::Dictionary::decode_dict(&dict).expect("a valid dictionary"); + let text = String::from_utf8_lossy(&dict); + assert!(text.contains("After=network-online.target")); + assert!(text.contains("Restart=on-failure")); +} + +/// The legacy trainer is handed its samples as the reference's command hands +/// them over: each file one sample of at most 128 KiB, or cut whole into `-B` +/// pieces; empty files left out; fewer than five samples refused. +#[test] +fn training_samples_are_loaded_the_way_the_reference_loads_them() { + let dir = Scratch::new("training-load"); + let big = dir.file("big", &vec![b'x'; 300 << 10]); + let small = dir.file("small", b"0123456789"); + let empty = dir.file("empty", b""); + let inputs = vec![big.clone(), small.clone(), empty.clone()]; + + assert!( + load_training_samples(&inputs, None, None).is_err(), + "two samples are too few" + ); + + let cut = load_training_samples(&inputs, Some(64 << 10), None).unwrap(); + let mut sizes = cut.sizes.clone(); + sizes.sort_unstable(); + assert_eq!( + sizes, + vec![10, 44 << 10, 64 << 10, 64 << 10, 64 << 10, 64 << 10] + ); + assert_eq!(cut.corpus.len(), (300 << 10) + 10, "whole files once cut"); + + let mut many = Vec::new(); + for i in 0..5 { + many.push(dir.file(&format!("big{i}"), &vec![b'y'; 200 << 10])); + } + let capped = load_training_samples(&many, None, None).unwrap(); + assert_eq!( + capped.sizes, + vec![128 << 10; 5], + "each file capped at 128 KiB" + ); + let limited = load_training_samples(&many, None, Some(300 << 10)).unwrap(); + assert_eq!(limited.sizes.len(), 2, "-M bounds what is loaded"); +} + +/// The files are taken in the reference's shuffled order (`DiB_shuffle`), so +/// the same file list lays the corpus out the same way; the legacy dictionary +/// trained from a fixed list then carries the reference's content byte for +/// byte. The order is what `dibio.c`'s own `DiB_shuffle`, compiled as is, +/// makes of eight entries. +#[test] +fn training_files_are_shuffled_like_the_reference_shuffles_them() { + let mut order: Vec = (0..8).collect(); + shuffle_training_files(&mut order); + assert_eq!(order, vec![4, 5, 2, 0, 6, 1, 7, 3]); } /// A window is a promise about how much memory decoding will need, so it is @@ -1879,6 +2084,79 @@ fn benchmark_flags_parse_level_range() { assert_eq!(opts.bench_end, opts.bench_start); } +/// `-B#` is read the way the reference reads it (a count with `K` / `M`), and +/// zero is no block size at all. +#[test] +fn block_size_is_read_like_the_reference_reads_it() { + assert_eq!( + parse(&["-b", "-B64K", "f"]).unwrap().block_size, + Some(64 << 10) + ); + assert_eq!( + parse(&["-b", "-B1MiB", "f"]).unwrap().block_size, + Some(1 << 20) + ); + assert_eq!( + parse(&["-b", "-B4096", "f"]).unwrap().block_size, + Some(4096) + ); + assert_eq!(parse(&["-b", "-B0", "f"]).unwrap().block_size, None); + assert_eq!(parse(&["-b", "-B", "f"]).unwrap().block_size, None); + assert!(parse(&["-b", "-B1G", "f"]).is_err(), "no G multiplier"); + assert!(parse(&["-b", "-Bx", "f"]).is_err()); +} + +/// `--block-size=#` is the long spelling of `-B#`, as in the reference, which +/// reads both into the one setting the benchmark and the trainers cut by. +#[test] +fn the_long_block_size_sets_what_b_sets() { + assert_eq!( + parse(&["-b", "--block-size=64K", "f"]).unwrap().block_size, + Some(64 << 10) + ); + assert_eq!( + parse(&["-b", "--block-size", "4096", "f"]) + .unwrap() + .block_size, + Some(4096) + ); + assert_eq!( + parse(&["-b", "--block-size=0", "f"]).unwrap().block_size, + None, + "zero is no block size, as for -B0" + ); + assert!(parse(&["-b", "--block-size=x", "f"]).is_err()); +} + +/// The benchmark compresses every input as frames of its own, cut into `-B` +/// pieces from 32 bytes up, as the reference's block table does; an empty +/// input yields no frame, and a smaller `-B` cuts nothing. +#[test] +fn benchmark_frames_follow_the_inputs_and_the_block_size() { + let bench_chunk_lengths = + |sizes: &[u64], block| super::bench_chunk_lengths(sizes, block).collect::>(); + assert_eq!(bench_chunk_lengths(&[100, 50], None), vec![100, 50]); + assert_eq!( + bench_chunk_lengths(&[100, 50], Some(40)), + vec![40, 40, 20, 40, 10] + ); + assert_eq!( + bench_chunk_lengths(&[100, 0, 50], Some(64)), + vec![64, 36, 50] + ); + assert_eq!( + bench_chunk_lengths(&[100], Some(31)), + vec![100], + "below 32 is ignored" + ); + assert_eq!(bench_chunk_lengths(&[100], Some(32)), vec![32, 32, 32, 4]); + assert_eq!( + bench_frames_extent(&[80], Some(40)), + Some((2 * structured_zstd::encoding::compress_bound(40) as u64, 40)), + "each frame pays its own framing" + ); +} + #[test] fn dash_is_a_stdin_input() { let opts = parse(&["-d", "-"]).unwrap(); @@ -3352,6 +3630,69 @@ fn an_unreadable_directory_named_alone_fails_the_run() { } } +/// `--show-default-cparams` prints the reference command's layout: the name and +/// size, then one line per parameter, the strategy by upstream's name and +/// ordinal. Level 3 on an unknown-size source is the `clevels.h` row +/// `{21, 16, 17, 1, 5, 0, ZSTD_dfast}` unadjusted. +#[test] +fn show_default_cparams_prints_the_reference_layout() { + let mut out = Vec::new(); + write_default_cparams(&mut out, STDIN_MARK, None, 0, 3).unwrap(); + assert_eq!( + String::from_utf8(out).unwrap(), + "/*stdin*\\ (src size unknown)\n \ + - windowLog : 21\n \ + - chainLog : 16\n \ + - hashLog : 17\n \ + - searchLog : 1\n \ + - minMatch : 5\n \ + - targetLength : 0\n \ + - strategy : ZSTD_dfast (2)\n" + ); +} + +/// An empty file is reported with its size, zero, but sized as an unknown +/// source, since the reference reads a zero size as unknown here; a real size +/// moves the selection (level 11 on 4 KiB is the optimal parser). +#[test] +fn show_default_cparams_sizes_an_empty_file_as_unknown() { + let mut empty = Vec::new(); + write_default_cparams(&mut empty, "empty", Some(0), 0, 11).unwrap(); + let mut unknown = Vec::new(); + write_default_cparams(&mut unknown, "empty", None, 0, 11).unwrap(); + let empty = String::from_utf8(empty).unwrap(); + let unknown = String::from_utf8(unknown).unwrap(); + assert!(empty.starts_with("empty (0 bytes)\n"), "{empty}"); + assert_eq!( + empty.lines().skip(1).collect::>(), + unknown.lines().skip(1).collect::>(), + "same parameters as an unknown size" + ); + + let mut small = Vec::new(); + write_default_cparams(&mut small, "small", Some(4096), 0, 11).unwrap(); + let small = String::from_utf8(small).unwrap(); + assert!( + small.contains(" - strategy : ZSTD_btopt (7)\n"), + "{small}" + ); + assert!(small.contains(" - windowLog : 12\n"), "{small}"); +} + +/// Decompression has no parameters to show, so the flag is refused there, as +/// the reference command refuses it. +#[test] +fn show_default_cparams_is_refused_when_decompressing() { + let scratch = Scratch::new("cparamsd"); + let frame = scratch.file("f.zst", &frame_of(b"payload")); + let mut opts = parse(&["-d", "-q", "--show-default-cparams", "f"]).unwrap(); + opts.inputs = vec![frame]; + let err = run(opts) + .expect_err("decompression with --show-default-cparams is refused") + .to_string(); + assert!(err.contains("decompression mode"), "{err}"); +} + /// Decompression reports how many bytes came out, which is what `-t` and the /// summaries print; a corrupted checksum is ignored under `--no-check`. #[test] @@ -4131,6 +4472,96 @@ fn advanced_parameters_reach_the_frame() { ); } +/// `--max` sets every knob to its hardest end, as the reference's +/// `setMaxCompression` does, with the window stopped where this build still +/// decodes. It unlocks the ultra levels and long-distance matching, replaces a +/// `--zstd=` list given before it, and is adjusted by one given after it. +#[test] +#[cfg(target_pointer_width = "64")] +fn max_sets_every_knob_to_its_hardest_end() { + let opts = parse(&["--max", "f"]).unwrap(); + assert!(opts.long, "--max enables long-distance matching"); + assert_eq!( + opts.advanced, + AdvancedParams { + window_log: Some(27), + chain_log: Some(30), + hash_log: Some(30), + search_log: Some(30), + min_match: Some(3), + target_length: Some(131_072), + strategy: Some(Strategy::Btultra2), + ldm_hash_log: Some(30), + ldm_min_match: Some(16), + ldm_bucket_size_log: Some(8), + ldm_hash_rate_log: None, + } + ); + assert_eq!( + parse(&["--max", "-22", "f"]).unwrap().level, + 22, + "--max unlocks the ultra levels" + ); + assert!( + parse(&["-3", "--max", "f"]).is_ok(), + "the strategy it sets carries long-distance matching at any level" + ); + + let before = parse(&["--zstd=wlog=20,hlog=18", "--max", "f"]).unwrap(); + assert_eq!( + before.advanced, + max_compression_params(), + "an earlier list is replaced" + ); + let after = parse(&["--max", "--zstd=wlog=20", "f"]).unwrap(); + assert_eq!( + after.advanced.window_log, + Some(20), + "a later list adjusts it" + ); + assert_eq!( + after.advanced.chain_log, + Some(30), + "and leaves the rest at the maximum" + ); +} + +/// A `--max` frame over a known-size input is down-sized to the input, so it +/// compresses without the widest tables and decodes back to the input. +#[test] +#[cfg(target_pointer_width = "64")] +fn a_max_frame_round_trips() { + let opts = parse(&["--max", "f"]).unwrap(); + // Small: at a search depth of 2^30 a debug build walks every candidate. + let payload: Vec = (0..16 * 1024u32) + .map(|i| b'a' + (i.wrapping_mul(2_654_435_761) >> 28) as u8) + .collect(); + let mut frame = Vec::new(); + compress_stream( + payload.as_slice(), + &mut frame, + &FrameSettings { + pledged_size: Some(payload.len() as u64), + ..FrameSettings::from_options(&opts) + }, + &mut no_dict(), + ) + .unwrap(); + assert!(frame.len() < payload.len(), "the payload is compressible"); + assert_eq!(decoded(&frame).unwrap(), payload); +} + +/// On a 32-bit target `--max` is refused, as the reference refuses it: its +/// tables at their widest do not fit the address space. +#[test] +#[cfg(not(target_pointer_width = "64"))] +fn max_is_refused_on_a_32_bit_target() { + let Err(err) = parse(&["--max", "f"]) else { + panic!("--max must be refused on a 32-bit target"); + }; + assert!(err.to_string().contains("32-bit"), "{err}"); +} + /// `--long` below level 16 is refused because the matcher does not run there, /// unless `--zstd=strat=` moves the level onto a parser where it does. #[test] @@ -4275,7 +4706,10 @@ fn trainer_parameters_parse_and_build_options() { let opts = parse(&["--train-cover", "s1"]).unwrap(); assert_eq!(opts.trainer, Trainer::Cover); assert!(opts.trainer_params.is_default()); - assert!(parse(&["--train-legacy", "s1"]).is_err()); + assert_eq!( + parse(&["--train-legacy", "s1"]).unwrap().trainer, + Trainer::Legacy + ); } /// An empty `--filelist` is nothing to do for the modes that stream, but the diff --git a/zstd/src/decoding/prefetch.rs b/zstd/src/decoding/prefetch.rs index 4d7525b7c..d497b5666 100644 --- a/zstd/src/decoding/prefetch.rs +++ b/zstd/src/decoding/prefetch.rs @@ -119,13 +119,15 @@ fn prefetch_stride_x86(slice: &[u8]) { } } -#[cfg(target_arch = "aarch64")] +// The aarch64 hints are inline assembly, which Miri cannot execute; under Miri +// they take the no-op fallback below, a hint having no observable effect. +#[cfg(all(target_arch = "aarch64", not(miri)))] #[inline(always)] fn prefetch_slice_impl_l1(slice: &[u8]) { prefetch_stride_aarch64::(slice); } -#[cfg(target_arch = "aarch64")] +#[cfg(all(target_arch = "aarch64", not(miri)))] #[inline(always)] fn prefetch_first_line_l1_impl(ptr: *const u8) { use core::arch::asm; @@ -138,13 +140,13 @@ fn prefetch_first_line_l1_impl(ptr: *const u8) { } } -#[cfg(target_arch = "aarch64")] +#[cfg(all(target_arch = "aarch64", not(miri)))] #[inline(always)] fn prefetch_slice_impl_t1(slice: &[u8]) { prefetch_stride_aarch64::(slice); } -#[cfg(target_arch = "aarch64")] +#[cfg(all(target_arch = "aarch64", not(miri)))] #[inline(always)] fn prefetch_stride_aarch64(slice: &[u8]) { use core::arch::asm; @@ -182,7 +184,7 @@ fn prefetch_stride_aarch64(slice: &[u8]) { #[cfg(not(any( target_arch = "x86_64", all(target_arch = "x86", target_feature = "sse"), - target_arch = "aarch64", + all(target_arch = "aarch64", not(miri)), )))] #[inline(always)] fn prefetch_slice_impl_l1(_slice: &[u8]) {} @@ -190,7 +192,7 @@ fn prefetch_slice_impl_l1(_slice: &[u8]) {} #[cfg(not(any( target_arch = "x86_64", all(target_arch = "x86", target_feature = "sse"), - target_arch = "aarch64", + all(target_arch = "aarch64", not(miri)), )))] #[inline(always)] fn prefetch_first_line_l1_impl(_ptr: *const u8) {} @@ -198,7 +200,7 @@ fn prefetch_first_line_l1_impl(_ptr: *const u8) {} #[cfg(not(any( target_arch = "x86_64", all(target_arch = "x86", target_feature = "sse"), - target_arch = "aarch64", + all(target_arch = "aarch64", not(miri)), )))] #[inline(always)] fn prefetch_slice_impl_t1(_slice: &[u8]) {} diff --git a/zstd/src/dictionary/legacy.rs b/zstd/src/dictionary/legacy.rs new file mode 100644 index 000000000..27f68ce2e --- /dev/null +++ b/zstd/src/dictionary/legacy.rs @@ -0,0 +1,706 @@ +//! The reference's original dictionary trainer (`zdict.c`, +//! `ZDICT_trainFromBuffer_legacy`), which `zstd --train-legacy` runs. +//! +//! It walks the suffix array of the whole corpus and, for every position not +//! yet covered, measures how many other positions share a prefix of at least +//! [`MIN_MATCH_LENGTH`] bytes with it. A prefix repeated often enough becomes a +//! candidate segment, scored by the bytes it would save; overlapping candidates +//! merge, and the best of them, up to the requested size, are the dictionary. +//! Selectivity sets how often "often enough" is: a candidate needs at least +//! `samples >> selectivity` repetitions, and never fewer than [`MIN_RATIO`]. + +use alloc::vec; +use alloc::vec::Vec; + +use super::suffix_array::suffix_array; +use crate::encoding::fastpath::{ + FastpathKernel, dispatch_common_prefix_len_ptr_with_kernel, select_kernel, +}; + +/// Fewest repetitions that make a prefix a candidate (`MINRATIO`). +const MIN_RATIO: u32 = 4; +/// Shortest prefix counted as a repetition (`MINMATCHLENGTH`). +const MIN_MATCH_LENGTH: usize = 7; +/// Longest repetition length told apart when scoring (`LLIMIT`). +const LENGTH_LIMIT: usize = 64; +/// Smallest candidate table (`DICTLISTSIZE_DEFAULT`). +const DICT_LIST_SIZE_DEFAULT: usize = 10_000; +/// Bytes of noise after the corpus, so a comparison running off its end stops +/// on a mismatch (`NOISELENGTH`). +const NOISE_LENGTH: usize = 32; +/// Most corpus the trainer reads; whole samples beyond it are dropped +/// (`ZDICT_MAX_SAMPLES_SIZE`). +const MAX_SAMPLES_SIZE: usize = 2000 << 20; +/// Smallest dictionary content the trainer returns (`ZDICT_CONTENTSIZE_MIN`). +const CONTENT_SIZE_MIN: usize = 128; +/// Smallest corpus worth training on (`ZDICT_MIN_SAMPLES_SIZE`). +pub(crate) const MIN_SAMPLES_SIZE: usize = CONTENT_SIZE_MIN * MIN_RATIO as usize; +/// Selectivity the reference uses when none is given (`g_selectivity_default`). +pub const DEFAULT_SELECTIVITY: u32 = 9; + +/// Smallest dictionary the trainer is asked for (`ZDICT_DICTSIZE_MIN`). +pub(crate) const DICT_SIZE_MIN: usize = 256; + +/// What was too small for the legacy trainer to produce a dictionary. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum TooSmall { + /// The requested size is below [`DICT_SIZE_MIN`]. + Dictionary, + /// The corpus is smaller than [`MIN_SAMPLES_SIZE`]. + Corpus, + /// The corpus repeats too little to fill [`CONTENT_SIZE_MIN`] bytes. + Content, +} + +/// A candidate segment (`dictItem`). In the table, entry 0 is a header whose +/// `pos` counts the entries in use, the header included, and whose `savings` +/// is the largest value, so the sorted insert stops on it. +#[derive(Debug, Clone, Copy, Default)] +struct DictItem { + pos: u32, + length: u32, + savings: u32, +} + +/// The reference's noise band (`ZDICT_fillNoise`), the same bytes on every run. +fn noise_band() -> [u8; NOISE_LENGTH] { + let mut band = [0u8; NOISE_LENGTH]; + let mut acc: u32 = 2_654_435_761; + for byte in &mut band { + acc = acc.wrapping_mul(2_246_822_519); + *byte = (acc >> 21) as u8; + } + band +} + +/// The corpus followed by its noise band, read through bounds-checked +/// accessors that treat anything past the band as a mismatch. +/// +/// The samples are borrowed and the band kept apart, and the accessors join +/// the two. The reference copies the whole corpus into a buffer one band +/// longer (`ZDICT_trainFromBuffer_legacy`), a second copy of the samples while +/// the caller still holds them for the finalizer. +struct Corpus<'a> { + samples: &'a [u8], + noise: [u8; NOISE_LENGTH], + /// The compare kernel for this CPU, resolved once for the whole run. + kernel: FastpathKernel, +} + +impl<'a> Corpus<'a> { + fn new(samples: &'a [u8]) -> Self { + Self { + samples, + noise: noise_band(), + kernel: select_kernel(), + } + } + + /// Length of the samples and the band together. + #[inline] + fn len(&self) -> usize { + self.samples.len() + NOISE_LENGTH + } + + #[inline] + fn byte(&self, at: usize) -> Option { + match self.samples.get(at) { + Some(&byte) => Some(byte), + None => self.noise.get(at - self.samples.len()).copied(), + } + } + + /// `N` bytes from `at`, or `None` past the band. + #[inline(always)] + fn read(&self, at: usize) -> Option<[u8; N]> { + if let Some(bytes) = self.samples.get(at..at + N) { + return Some(bytes.try_into().expect("N bytes")); + } + self.read_across(at) + } + + /// [`Self::read`] for a read that reaches the band, which only the last + /// few positions of the corpus make. + #[cold] + #[inline(never)] + fn read_across(&self, at: usize) -> Option<[u8; N]> { + let mut out = [0u8; N]; + for (i, slot) in out.iter_mut().enumerate() { + *slot = self.byte(at + i)?; + } + Some(out) + } + + #[inline] + fn read16(&self, at: usize) -> Option { + self.read::<2>(at).map(u16::from_le_bytes) + } + + #[inline] + fn read64(&self, at: usize) -> Option { + self.read::<8>(at).map(u64::from_le_bytes) + } + + /// Bytes `a` and `b` have in common (`ZDICT_count`): through the CPU's + /// vector compare while both runs lie in the samples, where the corpus' + /// long repeats make the comparison most of the analysis, then a byte at + /// a time across into the band. + #[inline] + fn common(&self, a: usize, b: usize) -> usize { + let samples = self.samples; + // Every position compared is a suffix, below the samples' length, or + // the noise slot, at it; one at the slot compares nothing in the + // samples and goes straight to the band. + debug_assert!(a.max(b) <= samples.len()); + let in_samples = samples.len() - a.max(b); + let base = samples.as_ptr(); + // SAFETY: `a, b <= max(a, b)` and `in_samples == samples.len() - + // max(a, b)`, so both runs of `in_samples` bytes lie in the samples. + let n = unsafe { + dispatch_common_prefix_len_ptr_with_kernel( + self.kernel, + base.add(a), + base.add(b), + in_samples, + ) + }; + if n < in_samples { + return n; + } + self.common_tail(a, b, n) + } + + /// The bytes of [`Self::common`] from `n` on, where the shorter run has + /// reached the end of the samples: on into the band. + #[cold] + #[inline(never)] + fn common_tail(&self, a: usize, b: usize, mut n: usize) -> usize { + let Some(limit) = self.len().checked_sub(a.max(b)) else { + return 0; + }; + while n < limit && self.byte(a + n) == self.byte(b + n) { + n += 1; + } + n + } + + /// Copy the `out.len()` bytes from `from` into `out`. + fn copy_to(&self, from: usize, out: &mut [u8]) { + if let Some(bytes) = self.samples.get(from..from + out.len()) { + out.copy_from_slice(bytes); + return; + } + for (i, slot) in out.iter_mut().enumerate() { + *slot = self + .byte(from + i) + .expect("a segment lies inside the corpus"); + } + } +} + +/// How many text positions ahead the analysis warms the suffix-array line +/// around a position's rank. +const FAR_AHEAD: usize = 16; + +/// How many text positions ahead the analysis warms the text of a position's +/// two neighbours in suffix order. +const NEAR_AHEAD: usize = 8; + +/// The suffix array with one extra slot on each side, both pointing into the +/// noise band, as the reference lays it out (`suffix0[0]` and +/// `suffix[bufferSize]`): a walk off either end compares against noise and +/// stops there. +struct Suffixes { + sa: Vec, + noise: usize, +} + +impl Suffixes { + fn new(sa: Vec, len: usize) -> Self { + Self { sa, noise: len } + } + + /// Ranks in the array; rank `ranks()` is the upper noise slot. + #[inline] + fn ranks(&self) -> i64 { + self.sa.len() as i64 + } + + /// `suffix[at]`, where ranks `-1` and `len` are the noise band (the walks + /// stop at those two slots). A negative rank wraps past every real one, + /// so a single compare tells the two apart. + #[inline] + fn at(&self, at: i64) -> usize { + match self.sa.get(at as u64 as usize) { + Some(&pos) => pos as usize, + None => self.noise, + } + } + + /// Warm what the analysis of text position `cursor` will read, ahead of + /// reaching it: its rank is random, so the array around that rank and the + /// text of the two neighbours there miss the cache every time. The array + /// line is warmed for the position [`FAR_AHEAD`] on, the neighbours' text + /// for the one [`NEAR_AHEAD`] on, whose array line that earlier hint has + /// brought in. A hint only; ranks at either end read a clamped slot. + #[inline(always)] + fn prefetch_for(&self, rank: &[u32], samples: &[u8], cursor: usize) { + use crate::decoding::prefetch::prefetch_l1_at; + let len = self.sa.len(); + if let Some(&far) = rank.get(cursor + FAR_AHEAD) { + prefetch_l1_at(self.sa.as_ptr().wrapping_add(far as usize).cast()); + } + if let Some(&near) = rank.get(cursor + NEAR_AHEAD) { + let near = near as usize; + // Rank 0 wraps past the end and clamps with the top rank. + let below = near.wrapping_sub(1).min(len - 1); + let above = (near + 1).min(len - 1); + // SAFETY: both are clamped below `len == self.sa.len()`. + let (below, above) = unsafe { + ( + *self.sa.get_unchecked(below) as usize, + *self.sa.get_unchecked(above) as usize, + ) + }; + prefetch_l1_at(samples.as_ptr().wrapping_add(below)); + prefetch_l1_at(samples.as_ptr().wrapping_add(above)); + } + } +} + +/// Train dictionary content from `samples`, the concatenation of samples whose +/// lengths are `sample_sizes`, keeping at most `dict_size` bytes. +/// +/// The segments are laid out best last, so the most valuable content sits +/// nearest the data and is reached with the smallest offsets. `selectivity` of +/// zero is [`DEFAULT_SELECTIVITY`]. +pub(crate) fn train_legacy_raw( + samples: &[u8], + sample_sizes: &[usize], + dict_size: usize, + selectivity: u32, +) -> Result, TooSmall> { + let total: usize = sample_sizes.iter().sum(); + debug_assert_eq!(total, samples.len(), "the sizes describe the corpus"); + if dict_size < DICT_SIZE_MIN { + return Err(TooSmall::Dictionary); + } + if total < MIN_SAMPLES_SIZE { + return Err(TooSmall::Corpus); + } + let selectivity = if selectivity == 0 { + DEFAULT_SELECTIVITY + } else { + selectivity + }; + let nb_samples = sample_sizes.len(); + let min_rep = if selectivity > 30 { + MIN_RATIO + } else { + // A repetition count past u32 is more than a u32-indexed corpus has + // positions, so nothing could qualify. + u32::try_from(nb_samples >> selectivity).map_err(|_| TooSmall::Content)? + }; + let list_size = DICT_LIST_SIZE_DEFAULT.max(nb_samples).max(dict_size / 16); + let mut list = vec![DictItem::default(); list_size]; + list[0] = DictItem { + pos: 1, + length: 0, + savings: u32::MAX, + }; + + // Whole samples past the size limit are left out, last first. + let mut used = total; + let mut kept = nb_samples; + while used > MAX_SAMPLES_SIZE { + kept -= 1; + used -= sample_sizes[kept]; + } + let corpus = Corpus::new(&samples[..used]); + find_segments(&mut list, &corpus, used, min_rep); + + let segments = list[0].pos as usize; + let content_size: usize = list[1..segments].iter().map(|d| d.length as usize).sum(); + if content_size < CONTENT_SIZE_MIN { + return Err(TooSmall::Content); + } + // Keep the best segments that fit, in rank order. + let mut fit = 1; + let mut size = 0usize; + while fit < segments { + let next = size + list[fit].length as usize; + if next > dict_size { + break; + } + size = next; + fit += 1; + } + // Filled from the back, as the reference fills its buffer: rank 1 last. + let mut content = vec![0u8; size]; + let mut end = size; + for item in &list[1..fit] { + let length = item.length as usize; + let start = end - length; + let from = item.pos as usize; + corpus.copy_to(from, &mut content[start..end]); + end = start; + } + debug_assert_eq!(end, 0, "the kept segments fill the content exactly"); + Ok(content) +} + +/// `ZDICT_trainBuffer_legacy`: walk every uncovered position of the corpus in +/// text order and insert the segment its suffix neighbourhood yields. +fn find_segments(list: &mut [DictItem], corpus: &Corpus<'_>, len: usize, min_rep: u32) { + let min_ratio = min_rep.max(MIN_RATIO); + debug_assert_eq!(corpus.samples.len(), len); + let sa = suffix_array(corpus.samples); + let mut rank = vec![0u32; len]; + for (at, &pos) in sa.iter().enumerate() { + rank[pos as usize] = at as u32; + } + let suffixes = Suffixes::new(sa, len); + // Slack past the corpus, as the reference allocates it: a covered run may + // be marked into the noise band. + let mut done = vec![false; len + 16]; + let max_size = list.len() as u32; + + let mut cursor = 0usize; + while cursor < len { + suffixes.prefetch_for(&rank, corpus.samples, cursor); + if done[cursor] { + cursor += 1; + continue; + } + let solution = analyze_position( + &mut done, + &suffixes, + i64::from(rank[cursor]), + corpus, + min_ratio, + ); + if solution.length == 0 { + cursor += 1; + continue; + } + insert_item(list, max_size, solution, corpus); + cursor += solution.length as usize; + } +} + +/// Mark `at` covered, if it lies inside the marks. +#[inline] +fn mark(done: &mut [bool], at: usize) { + if let Some(slot) = done.get_mut(at) { + *slot = true; + } +} + +/// Mark `len` positions from `at` covered, as far as the marks reach. Both +/// are bounded by the corpus length, so the sum cannot overflow. +#[inline] +fn mark_run(done: &mut [bool], at: usize, len: usize) { + let end = (at + len).min(done.len()); + if at < end { + done[at..end].fill(true); + } +} + +/// `ZDICT_analyzePos`: the segment the suffix of rank `start` leads to, or an +/// empty one. Every position the analysis settles is marked in `done`. +fn analyze_position( + done: &mut [bool], + suffixes: &Suffixes, + mut start: i64, + corpus: &Corpus<'_>, + min_ratio: u32, +) -> DictItem { + let mut pos = suffixes.at(start); + let mut end = start; + let empty = DictItem::default(); + mark(done, pos); + + // A run of one repeated pair is skipped whole: it compresses without a + // dictionary. + let repeats = |a: usize, b: usize| matches!((corpus.read16(a), corpus.read16(b)), (Some(x), Some(y)) if x == y); + if repeats(pos, pos + 2) || repeats(pos + 1, pos + 3) || repeats(pos + 2, pos + 4) { + let pattern = corpus.read16(pos + 4); + let mut pattern_end = 6usize; + while pattern.is_some() && corpus.read16(pos + pattern_end) == pattern { + pattern_end += 2; + } + if corpus.byte(pos + pattern_end).is_some() + && corpus.byte(pos + pattern_end) == corpus.byte(pos + pattern_end - 1) + { + pattern_end += 1; + } + mark_run(done, pos + 1, pattern_end - 1); + return empty; + } + + // The neighbours sharing at least the minimum length, forward then back. + // Each walk stops at its noise slot as well as on a short match: every + // rank past the array reads the same band, so a corpus that repeats the + // band would otherwise walk forever. The reference reads past its two + // slots there, so it has no result to keep; on every other corpus the + // slot holds noise that ends the walk anyway. + loop { + end += 1; + if end >= suffixes.ranks() || corpus.common(pos, suffixes.at(end)) < MIN_MATCH_LENGTH { + break; + } + } + while start > 0 && corpus.common(pos, suffixes.at(start - 1)) >= MIN_MATCH_LENGTH { + start -= 1; + } + + if ((end - start) as u32) < min_ratio { + for id in start..end { + mark(done, suffixes.at(id)); + } + return empty; + } + + // Lengthen the shared prefix one byte at a time while the most common + // continuation still repeats often enough. + let mut refined_start = start; + let mut refined_end = end; + let mut mml = MIN_MATCH_LENGTH; + loop { + let mut current_char = 0u8; + let mut current_count = 0u32; + let mut current_id = refined_start; + let mut selected_count = 0u32; + let mut selected_id = current_id; + for id in refined_start..refined_end { + let c = corpus.byte(suffixes.at(id) + mml).unwrap_or(0); + if c != current_char { + if current_count > selected_count { + selected_count = current_count; + selected_id = current_id; + } + current_id = id; + current_char = c; + current_count = 0; + } + current_count += 1; + } + if current_count > selected_count { + selected_count = current_count; + selected_id = current_id; + } + if selected_count < min_ratio { + break; + } + refined_start = selected_id; + refined_end = refined_start + i64::from(selected_count); + mml += 1; + } + + // Measure the refined segment's neighbourhood. + start = refined_start; + pos = suffixes.at(refined_start); + end = start; + let mut lengths = [0u32; LENGTH_LIMIT]; + loop { + end += 1; + let length = corpus.common(pos, suffixes.at(end)).min(LENGTH_LIMIT - 1); + lengths[length] += 1; + // The upper noise slot is counted, as the reference counts it, and + // ends the walk (see the first walk above). + if length < MIN_MATCH_LENGTH || end >= suffixes.ranks() { + break; + } + } + { + let mut length = MIN_MATCH_LENGTH; + while length >= MIN_MATCH_LENGTH && start > 0 { + length = corpus + .common(pos, suffixes.at(start - 1)) + .min(LENGTH_LIMIT - 1); + lengths[length] += 1; + if length >= MIN_MATCH_LENGTH { + start -= 1; + } + } + } + + // The longest length still shared by enough neighbours. + let mut cumulative = [0u32; LENGTH_LIMIT]; + cumulative[LENGTH_LIMIT - 1] = lengths[LENGTH_LIMIT - 1]; + for i in (0..LENGTH_LIMIT - 1).rev() { + cumulative[i] = cumulative[i + 1] + lengths[i]; + } + let mut max_length = LENGTH_LIMIT - 1; + while max_length >= MIN_MATCH_LENGTH && cumulative[max_length] < min_ratio { + max_length -= 1; + } + // Do not end inside a run of the last byte. + { + let last = corpus.byte(pos + max_length - 1); + let mut l = max_length; + while l >= 2 && corpus.byte(pos + l - 2) == last { + l -= 1; + } + max_length = l; + } + if max_length < MIN_MATCH_LENGTH { + return empty; + } + + let mut savings = [0u32; LENGTH_LIMIT]; + for i in MIN_MATCH_LENGTH..=max_length { + savings[i] = savings[i - 1].wrapping_add(lengths[i].wrapping_mul((i - 3) as u32)); + } + let solution = DictItem { + pos: pos as u32, + length: max_length as u32, + savings: savings[max_length], + }; + + for id in start..end { + let tested = suffixes.at(id); + let length = if tested == pos { + max_length + } else { + corpus.common(pos, tested).min(max_length) + }; + mark_run(done, tested, length); + } + solution +} + +/// Whether the `length` bytes at `a` equal those at `b` (`isIncluded`); both +/// runs must lie inside the corpus. Compares `length` bytes and no more, where +/// `common` would run on to the end of the shared prefix. +fn is_included(corpus: &Corpus<'_>, a: usize, b: usize, length: usize) -> bool { + let samples = corpus.samples; + if let (Some(x), Some(y)) = (samples.get(a..a + length), samples.get(b..b + length)) { + return x == y; + } + a.max(b) + length <= corpus.len() + && (0..length).all(|i| corpus.byte(a + i) == corpus.byte(b + i)) +} + +/// Move entry `at` towards the front while its savings beat its predecessor's. +fn promote(list: &mut [DictItem], mut at: usize) -> usize { + let item = list[at]; + while at > 1 && list[at - 1].savings < item.savings { + list[at] = list[at - 1]; + at -= 1; + } + list[at] = item; + at +} + +/// `ZDICT_tryMerge`: fold `elt` into an entry it overlaps, skipping entry +/// `skip`. Returns the merged entry's index, or 0 when nothing merged. +fn try_merge(list: &mut [DictItem], elt: DictItem, skip: usize, corpus: &Corpus<'_>) -> usize { + let size = list[0].pos as usize; + let elt_end = elt.pos + elt.length; + + // An existing entry starts inside `elt`: extend it backwards. The scans + // walk the table as a slice, and the rare skipped entry is tested last. + let mut hit = 0; + for (u, item) in list[..size].iter().enumerate().skip(1) { + if item.pos > elt.pos && item.pos <= elt_end && u != skip { + hit = u; + break; + } + } + if hit != 0 { + let item = &mut list[hit]; + let added = item.pos - elt.pos; + item.length += added; + item.pos = elt.pos; + item.savings = item + .savings + .wrapping_add(elt.savings.wrapping_mul(added) / elt.length); + item.savings = item.savings.wrapping_add(elt.length / 8); + return promote(list, hit); + } + + // `elt` starts inside an existing entry, or right after a copy of it. The + // eight bytes after `elt`'s start are the same for every entry, so they + // are read once rather than per entry as the reference reads them. + let elt_head = corpus.read64(elt.pos as usize + 1); + for u in 1..size { + let item = list[u]; + if u == skip { + continue; + } + if item.pos + item.length >= elt.pos && item.pos < elt.pos { + let added = elt_end as i64 - i64::from(item.pos + item.length); + let item = &mut list[u]; + item.savings = item.savings.wrapping_add(elt.length / 8); + if added > 0 { + item.length += added as u32; + item.savings = item + .savings + .wrapping_add(elt.savings.wrapping_mul(added as u32) / elt.length); + } + return promote(list, u); + } + if elt_head.is_some() + && corpus.read64(item.pos as usize) == elt_head + && is_included( + corpus, + item.pos as usize, + elt.pos as usize + 1, + item.length as usize, + ) + { + // The reference takes this product at pointer width, where it does + // not wrap, unlike the two above. + let added = (i64::from(elt.length) - i64::from(item.length)).max(1) as u64; + let item = &mut list[u]; + item.pos = elt.pos; + item.savings = item + .savings + .wrapping_add((u64::from(elt.savings) * added / u64::from(elt.length)) as u32); + item.length = elt.length.min(item.length + 1); + return u; + } + } + 0 +} + +/// `ZDICT_removeDictItem`. +fn remove_item(list: &mut [DictItem], id: usize) { + if id == 0 { + return; + } + let max = list[0].pos as usize; + for u in id..max - 1 { + list[u] = list[u + 1]; + } + list[0].pos -= 1; +} + +/// `ZDICT_insertDictItem`: merge `elt` into the table if it overlaps an entry, +/// and keep merging while the merged entry overlaps another; otherwise insert +/// it in savings order, dropping the last entry when the table is full. +fn insert_item(list: &mut [DictItem], max_size: u32, elt: DictItem, corpus: &Corpus<'_>) { + let mut merge_id = try_merge(list, elt, 0, corpus); + if merge_id != 0 { + loop { + let merged = try_merge(list, list[merge_id], merge_id, corpus); + if merged == 0 { + return; + } + remove_item(list, merge_id); + merge_id = merged; + } + } + let next = list[0].pos.min(max_size - 1) as usize; + let mut current = next - 1; + while list[current].savings < elt.savings { + list[current + 1] = list[current]; + current -= 1; + } + list[current + 1] = elt; + list[0].pos = next as u32 + 1; +} + +#[cfg(test)] +mod tests; diff --git a/zstd/src/dictionary/legacy/tests.rs b/zstd/src/dictionary/legacy/tests.rs new file mode 100644 index 000000000..8bf76c46e --- /dev/null +++ b/zstd/src/dictionary/legacy/tests.rs @@ -0,0 +1,138 @@ +use super::*; +use alloc::format; +use alloc::string::String; + +/// Every rank past either end of the suffix array reads the noise band, and +/// the band is the same bytes on every run. A corpus whose text repeats the +/// band therefore compared equal at every rank past the end, and the +/// neighbour walks never stopped: training hung on such a corpus. The walks +/// stop at the two noise slots the reference allocates. +#[test] +fn neighbour_walks_stop_at_the_noise_slots() { + // The corpus is the band itself, and its one suffix is the only rank in + // the array: every walk leaves the array on its first step. + let band = noise_band(); + let corpus = Corpus::new(&band); + let suffixes = Suffixes::new(vec![0], band.len()); + let mut done = vec![false; band.len() + 16]; + let solution = analyze_position(&mut done, &suffixes, 0, &corpus, MIN_RATIO); + // One suffix cannot repeat `MIN_RATIO` times. + assert_eq!(solution.length, 0); +} + +/// The same hang in the second walk, the one that measures the kept segment's +/// neighbourhood: four copies of the band repeat `MIN_RATIO` times, so the +/// analysis gets past the repetition check and walks the ranks again, reaching +/// the upper noise slot with the band still matching. +#[test] +fn the_measuring_walk_stops_at_the_noise_slot() { + let band = noise_band(); + let corpus_bytes: Vec = band + .iter() + .copied() + .cycle() + .take(4 * NOISE_LENGTH) + .collect(); + let corpus = Corpus::new(&corpus_bytes); + // In suffix order: each copy's suffix is a prefix of the one before it. + let suffixes = Suffixes::new(vec![96, 64, 32, 0], corpus_bytes.len()); + let mut done = vec![false; corpus_bytes.len() + 16]; + let solution = analyze_position(&mut done, &suffixes, 0, &corpus, MIN_RATIO); + assert!( + solution.length as usize >= MIN_MATCH_LENGTH, + "four copies repeat often enough to be kept" + ); +} + +/// `count` log lines of a few shapes, each line a sample. +fn log_samples(count: u32) -> (Vec, Vec) { + const SHAPES: [&str; 4] = [ + "ts={i} level=INFO msg=\"flush memtable\" tenant=demo table=orders region=eu-west\n", + "ts={i} level=WARN msg=\"slow compaction\" tenant=demo table=users region=us-east\n", + "ts={i} level=INFO msg=\"rotate segment\" tenant=acme table=orders region=eu-west\n", + "ts={i} level=ERROR msg=\"write stalled\" tenant=acme table=events region=ap-south\n", + ]; + let mut samples = Vec::new(); + let mut sizes = Vec::new(); + for i in 0..count { + let line = SHAPES[(i % 4) as usize].replace("{i}", &format!("{:08}", i * 7919)); + sizes.push(line.len()); + samples.extend_from_slice(line.as_bytes()); + } + (samples, sizes) +} + +/// The trainer keeps what the corpus repeats: every segment it returns is a +/// run of the corpus that occurs at least `MIN_RATIO` times, and the content +/// stays within the size asked for. +#[test] +fn the_content_is_made_of_repeated_corpus_runs() { + let (samples, sizes) = log_samples(400); + let content = train_legacy_raw(&samples, &sizes, 4096, 0).unwrap(); + assert!(content.len() >= CONTENT_SIZE_MIN && content.len() <= 4096); + let text = String::from_utf8_lossy(&content); + for field in [ + "tenant=demo table=orders", + "msg=\"flush memtable\"", + "region=eu-west", + ] { + assert!( + text.contains(field), + "{field} is repeated in every fourth sample" + ); + } + // A timestamp is unique per sample, so no whole one survives. + assert!(!text.contains(&format!("ts={:08}", 3 * 7919))); +} + +/// A size below the reference's minimum, a corpus under 512 bytes, and a corpus +/// with nothing repeated are refused, as `ZDICT_trainFromBuffer_legacy` refuses +/// them. +#[test] +fn too_little_to_train_on_is_refused() { + let (samples, sizes) = log_samples(400); + assert_eq!( + train_legacy_raw(&samples, &sizes, DICT_SIZE_MIN - 1, 0), + Err(TooSmall::Dictionary) + ); + let (small, small_sizes) = log_samples(5); + assert!(small.len() < MIN_SAMPLES_SIZE); + assert_eq!( + train_legacy_raw(&small, &small_sizes, 4096, 0), + Err(TooSmall::Corpus) + ); + let mut state = 0x9E37_79B9u32; + let noise: Vec = (0..8192) + .map(|_| { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + state as u8 + }) + .collect(); + assert_eq!( + train_legacy_raw(&noise, &[noise.len()], 4096, 0), + Err(TooSmall::Content) + ); +} + +/// Selectivity decides how often a segment must repeat: `samples >> s`, never +/// below four. With many samples a low selectivity demands more repetitions +/// than a rare shape has, so it keeps less than a high one. +#[test] +fn selectivity_sets_how_often_a_segment_must_repeat() { + let (mut samples, mut sizes) = log_samples(4096); + // A shape present in 40 samples: kept when 40 repetitions suffice. + for i in 0..40 { + let line = format!("rare shape {i:04} backup completed for volume vol-archive-7\n"); + sizes.push(line.len()); + samples.extend_from_slice(line.as_bytes()); + } + let strict = train_legacy_raw(&samples, &sizes, 64 * 1024, 6).unwrap(); + let loose = train_legacy_raw(&samples, &sizes, 64 * 1024, 9).unwrap(); + let has_rare = |content: &[u8]| String::from_utf8_lossy(content).contains("vol-archive-7"); + // 4136 >> 6 = 64 repetitions needed: the rare shape has 40. + assert!(!has_rare(&strict)); + // 4136 >> 9 = 8 repetitions needed. + assert!(has_rare(&loose)); +} diff --git a/zstd/src/dictionary/mod.rs b/zstd/src/dictionary/mod.rs index ef1ccde54..01f316165 100644 --- a/zstd/src/dictionary/mod.rs +++ b/zstd/src/dictionary/mod.rs @@ -25,7 +25,9 @@ mod cover; mod fastcover; mod frequency; +mod legacy; mod reservoir; +mod suffix_array; use crate::bit_io::BitWriter; use crate::blocks::sequence_section::{ @@ -43,6 +45,7 @@ pub use fastcover::{ DEFAULT_D_CANDIDATES, DEFAULT_F_CANDIDATES, DEFAULT_K_CANDIDATES, FastCoverParams, FastCoverTuned, }; +pub use legacy::DEFAULT_SELECTIVITY; use std::{ boxed::Box, collections::{BinaryHeap, HashMap}, @@ -361,7 +364,10 @@ fn serialize_huffman_table(sample_data: &[u8], raw_content: &[u8]) -> io::Result } let mut table = HuffmanEncoderTable::build_from_data(stats.as_slice()); - if table.writeable_table_description_size().is_none() { + if table + .writeable_table_description_size(&mut crate::fse::fse_encoder::FSETable::blank()) + .is_none() + { // Sampled real data can land on the same shape: a flat alphabet wider // than 128 symbols. Fall back to the synthetic narrow one, which always // has a description. @@ -700,6 +706,94 @@ pub fn create_fastcover_dict_from_slice( Ok(tuned) } +/// Train and finalize a dictionary with the reference's original trainer, the +/// one `zstd --train-legacy` runs (`ZDICT_trainFromBuffer_legacy`). +/// +/// `samples` is every sample back to back and `sample_sizes` their lengths: +/// the trainer searches the corpus as a whole, and the number of samples sets +/// how often a segment has to repeat to be kept, `samples >> selectivity` +/// times and at least 4. A higher `selectivity` keeps more, rarer segments; +/// zero is [`DEFAULT_SELECTIVITY`]. Corpus past 2000 MiB is dropped a whole +/// sample at a time from the end. +/// +/// # Errors +/// +/// `InvalidInput` when `dict_size` is below 256 bytes, when the corpus is under +/// 512 bytes, when it repeats too little to yield 128 bytes of content, or when +/// `sample_sizes` does not add up to `samples.len()`. +/// +/// # Examples +/// +/// ``` +/// use structured_zstd::dictionary::{create_legacy_dict_from_slice, FinalizeOptions}; +/// +/// let mut samples = Vec::new(); +/// let mut sizes = Vec::new(); +/// for i in 0..200u32 { +/// let line = format!("tenant=demo table=orders key={i} region=eu status=shipped\n"); +/// sizes.push(line.len()); +/// samples.extend_from_slice(line.as_bytes()); +/// } +/// let mut dict = Vec::new(); +/// create_legacy_dict_from_slice(&samples, &sizes, &mut dict, 4096, 0, FinalizeOptions::default()) +/// .unwrap(); +/// assert!(dict.starts_with(&[0x37, 0xA4, 0x30, 0xEC])); +/// ``` +pub fn create_legacy_dict_from_slice( + samples: &[u8], + sample_sizes: &[usize], + output: &mut W, + dict_size: usize, + selectivity: u32, + finalize: FinalizeOptions, +) -> io::Result<()> { + let described = sample_sizes + .iter() + .try_fold(0usize, |total, &size| total.checked_add(size)); + if described != Some(samples.len()) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "the sample sizes do not add up to the corpus", + )); + } + let content = + legacy::train_legacy_raw(samples, sample_sizes, dict_size, selectivity).map_err(|err| { + let reason = match err { + legacy::TooSmall::Dictionary => format!( + "a legacy dictionary must be at least {} bytes", + legacy::DICT_SIZE_MIN + ), + legacy::TooSmall::Corpus => format!( + "the samples total {} bytes; the legacy trainer needs at least {}", + samples.len(), + legacy::MIN_SAMPLES_SIZE + ), + legacy::TooSmall::Content => { + "the samples repeat too little to yield dictionary content".into() + } + }; + io::Error::new(io::ErrorKind::InvalidInput, reason) + })?; + // Every sample, not only those the content search kept: the reference cuts + // the corpus to its size limit inside the search alone, for its suffix + // sort (zdict.c, `ZDICT_trainBuffer_legacy`), and builds the entropy tables + // from all of them (`ZDICT_trainFromBuffer_unsafe_legacy`). + let finalized = finalize_raw_dict(content.as_slice(), samples, dict_size, finalize)?; + output.write_all(finalized.as_slice()) +} + +/// The legacy trainer's content alone, for the reference comparison in +/// `ffi-bench`. +#[cfg(feature = "bench-internals")] +pub(crate) fn legacy_dict_content( + samples: &[u8], + sample_sizes: &[usize], + dict_size: usize, + selectivity: u32, +) -> Option> { + legacy::train_legacy_raw(samples, sample_sizes, dict_size, selectivity).ok() +} + /// Build a finalized FastCOVER dictionary, attach it to a fastest-level /// frame compressor, and compress a fresh payload. Returns /// `(finalized_dictionary, compressed_frame, original_payload)` so a diff --git a/zstd/src/dictionary/suffix_array.rs b/zstd/src/dictionary/suffix_array.rs new file mode 100644 index 000000000..69a2b2a4b --- /dev/null +++ b/zstd/src/dictionary/suffix_array.rs @@ -0,0 +1,438 @@ +//! Suffix-array construction by induced sorting (SA-IS, Nong, Zhang and Chan, +//! "Two Efficient Algorithms for Linear Time Suffix Array Construction"). +//! +//! A suffix array is unique for its text: the positions of every suffix in +//! lexicographic order, a suffix that is a prefix of another ordered first. +//! So any correct construction yields the array the reference trainer gets +//! from `divsufsort`, which is what makes the legacy trainer built on it +//! reproduce the reference's choices. + +use alloc::vec; +use alloc::vec::Vec; + +/// Marks an empty slot of the array under construction. +const EMPTY: u32 = u32::MAX; + +/// Below this length a comparison sort is cheaper than induction. +const NAIVE_THRESHOLD: usize = 10; + +/// A symbol of the text being sorted: the bytes at the top level, the ranks of +/// the reduced string in the recursion. +trait Symbol: Copy + Ord { + /// Whether the bucket table can outgrow the cache: the recursion's + /// alphabet is its number of names. + const WIDE: bool; + + fn index(self) -> usize; +} + +impl Symbol for u8 { + const WIDE: bool = false; + + #[inline] + fn index(self) -> usize { + usize::from(self) + } +} + +impl Symbol for u32 { + const WIDE: bool = true; + + #[inline] + fn index(self) -> usize { + self as usize + } +} + +/// How many entries ahead of the sweep its loads are prefetched. The text a +/// swept entry points into is prefetched twice as far ahead, so that the +/// symbol is in cache when the bucket cursor it selects is prefetched. +const PREFETCH_DISTANCE: usize = 32; + +/// Warm the loads the sweep makes when it reaches entries `near` and `far`: +/// the text before the suffix at `far` and, where the bucket table is large, +/// the cursor of the bucket the suffix at `near` selects. An index past the +/// array (a sweep's wrapped subtraction included) warms nothing. A hint only: +/// entries not yet written, or not live, aim the text prefetch anywhere and +/// read a clamped symbol. +#[inline(always)] +fn prefetch_ahead(s: &[T], sa: &[u32], buf: &[u32], near: usize, far: usize) { + let n = sa.len(); + if far < n { + // SAFETY: bounded by the test above. + let pos = unsafe { *sa.get_unchecked(far) } as usize; + crate::decoding::prefetch::prefetch_l1_at( + s.as_ptr().wrapping_add(pos.wrapping_sub(1)).cast(), + ); + } + if T::WIDE && near < n { + // SAFETY: bounded by the test above. + let near = unsafe { *sa.get_unchecked(near) } as usize; + // A dead entry (top bit set) or an empty one wraps past `n - 1`. + let at = near.wrapping_sub(1).min(n - 1); + // SAFETY: `at < n == s.len()`. + let c = unsafe { s.get_unchecked(at) }.index(); + crate::decoding::prefetch::prefetch_l1_at(buf.as_ptr().wrapping_add(c).cast()); + } +} + +/// The suffix array of `text`: `sa[i]` is the start of the `i`-th smallest +/// suffix. +/// +/// # Panics +/// +/// If `text` is 2 GiB or longer: positions are held as `u32` whose top bit +/// marks an entry during construction. +pub(crate) fn suffix_array(text: &[u8]) -> Vec { + assert!( + text.len() < 1 << 31, + "a suffix array of {} bytes does not fit 31-bit positions", + text.len() + ); + let mut sa = vec![0u32; text.len()]; + sa_is(text, &mut sa, usize::from(u8::MAX)); + sa +} + +/// Whether each suffix is S-type (smaller than the one after it), a bit per +/// position. The last is L-type against the virtual sentinel. +struct Types { + words: Vec, +} + +impl Types { + fn new(s: &[T]) -> Self { + let n = s.len(); + let mut words = vec![0u64; n.div_ceil(64)]; + let mut next = false; + for i in (0..n - 1).rev() { + let small = if s[i] == s[i + 1] { + next + } else { + s[i] < s[i + 1] + }; + words[i / 64] |= u64::from(small) << (i % 64); + next = small; + } + Self { words } + } + + #[inline] + fn is_s(&self, i: usize) -> bool { + (self.words[i / 64] >> (i % 64)) & 1 != 0 + } + + /// Whether `p` starts a leftmost S-type run. + #[inline] + fn is_lms(&self, p: usize, n: usize) -> bool { + p > 0 && p < n && self.is_s(p) && !self.is_s(p - 1) + } +} + +/// `buf[c]` = the first slot of symbol `c`'s bucket. +fn bucket_starts(counts: &[u32], buf: &mut [u32]) { + let mut sum = 0; + for (slot, &count) in buf.iter_mut().zip(counts) { + *slot = sum; + sum += count; + } +} + +/// `buf[c]` = one past the last slot of symbol `c`'s bucket. +fn bucket_ends(counts: &[u32], buf: &mut [u32]) { + let mut sum = 0; + for (slot, &count) in buf.iter_mut().zip(counts) { + sum += count; + *slot = sum; + } +} + +/// SA-IS over `s` into the first `s.len()` slots of `space`, whose symbols all +/// lie in `0..=upper`. The text is taken to end in a virtual sentinel smaller +/// than every symbol. Slots of `space` past the array are free for the +/// construction to use. +/// +/// Laid out as Yuta Mori's `sais.c` lays it out, with nothing beside the array +/// but the bucket tables and a bit per position: the reduced string is +/// gathered at the back of the array and sorted recursively into the front, so +/// no level holds a copy of its input or of the array. The bucket tables go in +/// the free slots when they fit, which in the recursion they usually do: a +/// level reduced to `m` symbols leaves `n - 2m` slots between the two. +fn sa_is(s: &[T], space: &mut [u32], upper: usize) { + let n = s.len(); + let (sa, free) = space.split_at_mut(n); + match n { + 0 => return, + 1 => { + sa[0] = 0; + return; + } + 2 => { + sa.copy_from_slice(if s[0] < s[1] { &[0, 1] } else { &[1, 0] }); + return; + } + _ => {} + } + if n < NAIVE_THRESHOLD { + for (at, slot) in sa.iter_mut().enumerate() { + *slot = at as u32; + } + sa.sort_unstable_by(|&a, &b| s[a as usize..].cmp(&s[b as usize..])); + return; + } + + let types = Types::new(s); + let k = upper + 1; + let mut owned: Vec; + let (counts, buf) = if free.len() >= 2 * k { + let (counts, rest) = free.split_at_mut(k); + counts.fill(0); + (counts, &mut rest[..k]) + } else { + owned = vec![0u32; 2 * k]; + owned.split_at_mut(k) + }; + for &c in s { + counts[c.index()] += 1; + } + let counts = &*counts; + + // Stage 1: the LMS positions at their buckets' ends, in any order, induce + // the order of the LMS substrings. + sa.fill(0); + bucket_ends(counts, buf); + let mut m = 0; + for p in (1..n).rev() { + if types.is_lms(p, n) { + let c = s[p].index(); + buf[c] -= 1; + sa[buf[c] as usize] = p as u32; + m += 1; + } + } + induce(s, sa, counts, buf); + if m == 0 { + // No LMS suffix: the induction from the last suffix alone is the + // whole order. + return; + } + + // Name each LMS substring by rank, equal substrings sharing a name. The + // sorted LMS positions are compacted to the front, each one's name goes + // to `m + p / 2` (LMS positions are at least two apart, so the slots are + // distinct and lie past the first `m`), then the names are gathered in + // text order at the back. A substring runs from its LMS position to the + // next one, or to the end of the text. + let mut k = 0; + for i in 0..n { + let v = sa[i] as usize; + if types.is_lms(v, n) { + sa[k] = v as u32; + k += 1; + } + } + debug_assert_eq!(k, m); + sa[m..].fill(EMPTY); + let substring_end = |p: usize| { + let mut end = p + 1; + while end < n && !types.is_lms(end, n) { + end += 1; + } + end + }; + let mut name = 0u32; + let mut prev = sa[0] as usize; + let mut prev_end = substring_end(prev); + debug_assert!(m + prev / 2 < n); + sa[m + prev / 2] = 0; + for i in 1..m { + let cur = sa[i] as usize; + let cur_end = substring_end(cur); + // Equal when as long and equal symbol for symbol, the symbol after + // included. One that runs into the end of the text ends at the virtual + // sentinel, which nothing else reaches, so it is unique. + let same = cur_end - cur == prev_end - prev + && cur_end < n + && prev_end < n + && s[cur..=cur_end] == s[prev..=prev_end]; + if !same { + name += 1; + } + debug_assert!(m + cur / 2 < n); + sa[m + cur / 2] = name; + prev = cur; + prev_end = cur_end; + } + let mut j = n; + for i in (m..n).rev() { + if sa[i] != EMPTY { + j -= 1; + sa[j] = sa[i]; + } + } + debug_assert_eq!(j, n - m); + + // Stage 2: sort the reduced string into the front of `sa`. With every + // name distinct its order is the names themselves, and no recursion is + // needed. + { + // `m <= n / 2`, so the front `m` slots and the back `m` are disjoint, + // and the recursion has the `n - 2m` between them to spare. + let (front, reduced) = sa.split_at_mut(n - m); + if name as usize + 1 == m { + for (at, &rank) in reduced.iter().enumerate() { + front[rank as usize] = at as u32; + } + } else { + sa_is(&*reduced, front, name as usize); + } + let order = &mut front[..m]; + // The reduced string is done with: its slots take the LMS positions + // in text order, which turn the sorted indices into positions. + let mut at = 0; + for p in 1..n { + if types.is_lms(p, n) { + reduced[at] = p as u32; + at += 1; + } + } + debug_assert_eq!(at, m); + for slot in order.iter_mut() { + *slot = reduced[*slot as usize]; + } + } + + // Stage 3: the sorted LMS positions to their buckets' ends, keeping their + // order, from the back so that no entry is overwritten before it is read + // (`sais.c`, `sais_main` stage 3). A bucket's end is at least the number + // of LMS positions of its symbol and below, so the write cursor never + // passes the read one. + bucket_ends(counts, buf); + let mut i = m; + let mut j = n; + while i > 0 { + let mut p = sa[i - 1]; + let c = s[p as usize].index(); + let end = buf[c] as usize; + while end < j { + j -= 1; + sa[j] = 0; + } + loop { + debug_assert!(j >= i); + j -= 1; + sa[j] = p; + i -= 1; + if i == 0 { + break; + } + p = sa[i - 1]; + if s[p as usize].index() != c { + break; + } + } + } + sa[..j].fill(0); + induce(s, sa, counts, buf); +} + +/// The two induction sweeps from the LMS positions already in `sa` (every +/// other slot zero): the L-type suffixes left to right from the bucket +/// starts, then the S-type ones right to left from the bucket ends. +/// +/// The sweeps are the whole cost of the construction: one random write per +/// suffix. They follow Yuta Mori's `sais.c` (`induceSA`): whether a suffix's +/// predecessor is to be induced in the sweep that reads it is decided when the +/// suffix is written, from the symbol before it, which is next to the one just +/// read, and kept as the complement of the position (`!j`, the top bit set). +/// The L sweep complements every entry it reads, which turns exactly the +/// entries whose predecessor is S-type into the live ones for the S sweep; the +/// S sweep restores the rest. The bucket cursor stays in a register while the +/// symbol does not change. +/// +/// Every index below is in bounds by the bucket layout: a position is below +/// `n`, a symbol lies in `0..=upper`, and each cursor stays inside the bucket +/// its symbol's suffixes fill. +fn induce(s: &[T], sa: &mut [u32], counts: &[u32], buf: &mut [u32]) { + let n = s.len(); + let live = |v: u32| (v as i32) > 0; + + bucket_starts(counts, buf); + let last = n - 1; + let mut c1 = s[last].index(); + let mut b = buf[c1] as usize; + sa[b] = if s[last - 1] < s[last] { + !(last as u32) + } else { + last as u32 + }; + b += 1; + for i in 0..n { + prefetch_ahead(s, sa, buf, i + PREFETCH_DISTANCE, i + 2 * PREFETCH_DISTANCE); + // SAFETY: `i < n == sa.len()`. + let v = unsafe { *sa.get_unchecked(i) }; + unsafe { *sa.get_unchecked_mut(i) = !v }; + if live(v) { + let j = v as usize - 1; + // SAFETY: `j < n == s.len()`. + let c0 = unsafe { s.get_unchecked(j) }.index(); + if c0 != c1 { + debug_assert!(c0 < buf.len() && c1 < buf.len()); + // SAFETY: both symbols lie in `0..=upper`. + unsafe { + *buf.get_unchecked_mut(c1) = b as u32; + b = *buf.get_unchecked(c0) as usize; + } + c1 = c0; + } + // `j` is L-type; its predecessor is live for this sweep when it is + // L-type as well, which here means not smaller. + let dead = j > 0 && unsafe { s.get_unchecked(j - 1) }.index() < c1; + debug_assert!(b < n); + // SAFETY: `b` stays inside bucket `c1`. + unsafe { *sa.get_unchecked_mut(b) = if dead { !(j as u32) } else { j as u32 } }; + b += 1; + } + } + + bucket_ends(counts, buf); + let mut c1 = 0usize; + let mut b = buf[0] as usize; + for i in (0..n).rev() { + prefetch_ahead( + s, + sa, + buf, + i.wrapping_sub(PREFETCH_DISTANCE), + i.wrapping_sub(2 * PREFETCH_DISTANCE), + ); + // SAFETY: `i < n == sa.len()`. + let v = unsafe { *sa.get_unchecked(i) }; + if live(v) { + let j = v as usize - 1; + // SAFETY: `j < n == s.len()`. + let c0 = unsafe { s.get_unchecked(j) }.index(); + if c0 != c1 { + debug_assert!(c0 < buf.len() && c1 < buf.len()); + // SAFETY: both symbols lie in `0..=upper`. + unsafe { + *buf.get_unchecked_mut(c1) = b as u32; + b = *buf.get_unchecked(c0) as usize; + } + c1 = c0; + } + // `j` is S-type; its predecessor is live when it is S-type as + // well, which here means not larger. + let dead = j == 0 || unsafe { s.get_unchecked(j - 1) }.index() > c1; + debug_assert!(b > 0); + b -= 1; + // SAFETY: `b` stays inside bucket `c1`. + unsafe { *sa.get_unchecked_mut(b) = if dead { !(j as u32) } else { j as u32 } }; + } else { + unsafe { *sa.get_unchecked_mut(i) = !v }; + } + } +} + +#[cfg(test)] +mod tests; diff --git a/zstd/src/dictionary/suffix_array/tests.rs b/zstd/src/dictionary/suffix_array/tests.rs new file mode 100644 index 000000000..cc15bccbf --- /dev/null +++ b/zstd/src/dictionary/suffix_array/tests.rs @@ -0,0 +1,71 @@ +use super::*; + +/// The array a comparison sort gives: the definition, not an algorithm. +fn naive(text: &[u8]) -> Vec { + let mut sa: Vec = (0..text.len() as u32).collect(); + sa.sort_by(|&a, &b| text[a as usize..].cmp(&text[b as usize..])); + sa +} + +fn lcg_bytes(seed: u64, len: usize, alphabet: u8) -> Vec { + let mut state = seed; + (0..len) + .map(|_| { + state = state + .wrapping_mul(6_364_136_223_846_793_005) + .wrapping_add(1_442_695_040_888_963_407); + ((state >> 33) % u64::from(alphabet)) as u8 + }) + .collect() +} + +/// Induced sorting agrees with the definition on the shapes that exercise its +/// branches: tiny texts (the direct cases), runs of one byte (all L-type, no +/// LMS positions), periodic text (equal LMS substrings, so the recursion runs +/// on repeated names), small and full alphabets. +#[test] +fn induced_sorting_matches_the_definition() { + let fixed: [&[u8]; 9] = [ + b"", + b"a", + b"ab", + b"ba", + b"banana", + b"mississippi", + b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + b"abababababababababababababababab", + b"abracadabra abracadabra abracadabra", + ]; + for text in fixed { + assert_eq!(suffix_array(text), naive(text), "text {text:?}"); + } + for (seed, alphabet) in [(1u64, 2u8), (2, 3), (3, 4), (4, 26), (5, 255)] { + for len in [9usize, 10, 11, 40, 257, 4096] { + let text = lcg_bytes(seed, len, alphabet); + assert_eq!( + suffix_array(&text), + naive(&text), + "len {len}, alphabet {alphabet}" + ); + } + } + let mut periodic = Vec::new(); + for _ in 0..300 { + periodic.extend_from_slice(b"tenant=demo key="); + } + assert_eq!(suffix_array(&periodic), naive(&periodic)); +} + +/// Short texts over two or three symbols, by the thousand: the shapes where an +/// LMS substring running into the end of the text has the length of another, +/// which naming has to tell apart without reading past the end, and where the +/// recursion goes several levels deep. +#[test] +fn many_short_texts_over_tiny_alphabets_sort_correctly() { + for seed in 0..2000u64 { + let alphabet = 2 + (seed % 2) as u8; + let len = 10 + (seed % 40) as usize; + let text = lcg_bytes(seed.wrapping_mul(0x9E37_79B9), len, alphabet); + assert_eq!(suffix_array(&text), naive(&text), "seed {seed}: {text:?}"); + } +} diff --git a/zstd/src/encoding/blocks/compressed.rs b/zstd/src/encoding/blocks/compressed.rs index b5ee79762..62750da4c 100644 --- a/zstd/src/encoding/blocks/compressed.rs +++ b/zstd/src/encoding/blocks/compressed.rs @@ -208,14 +208,58 @@ impl CompressedBlockScratch { struct SequencePrefixSums { lit: Vec, ml: Vec, + /// Each sequence's literal-length and match-length codes, and a prefix sum + /// of the extra bits the two carry. Neither depends on the offset history, + /// so they are derived once per block (upstream zstd `ZSTD_seqToCodes`) and + /// every split probe reads its range from here. + ll_code: Vec, + ml_code: Vec, + length_bits: Vec, +} + +/// The precomputed length codes of a run of sequences. +#[derive(Clone, Copy)] +struct LengthCodes<'a> { + ll: &'a [u8], + ml: &'a [u8], + /// Extra bits of every literal and match length in the run. + bits: usize, } impl SequencePrefixSums { fn heap_size(&self) -> usize { - (self.lit.capacity() + self.ml.capacity()) * core::mem::size_of::() + (self.lit.capacity() + self.ml.capacity() + self.length_bits.capacity()) + * core::mem::size_of::() + + self.ll_code.capacity() + + self.ml_code.capacity() + } + + /// The length codes of sequences `start..end`. + fn codes(&self, start: usize, end: usize) -> LengthCodes<'_> { + LengthCodes { + ll: &self.ll_code[start..end], + ml: &self.ml_code[start..end], + bits: self.length_bits[end] - self.length_bits[start], + } } fn rebuild(&mut self, sequences: &[RawSequence]) { + self.ll_code.clear(); + self.ml_code.clear(); + self.length_bits.clear(); + self.ll_code.reserve(sequences.len()); + self.ml_code.reserve(sequences.len()); + self.length_bits.reserve(sequences.len() + 1); + let mut bits = 0usize; + self.length_bits.push(0); + for seq in sequences { + let (ll, _, ll_bits) = encode_literal_length(seq.ll); + let (ml, _, ml_bits) = encode_match_len(seq.ml); + self.ll_code.push(ll); + self.ml_code.push(ml); + bits += ll_bits + ml_bits; + self.length_bits.push(bits); + } self.lit.clear(); self.ml.clear(); // `Vec::reserve_exact(additional)` adds `additional` elements ABOVE @@ -252,7 +296,7 @@ impl SequencePrefixSums { /// One collected sequence. /// -/// `off_base` holds the offset the matcher found until [`fill_wire_offsets`] +/// `off_base` holds the offset the matcher found until [`fill_and_count`] /// runs over the sequence, and the wire code from then on: 1/2/3 for the repeat /// offsets, N+3 for an explicit N. It is one field rather than two because the /// found offset has no reader once its code exists, and a fourth word would @@ -372,6 +416,7 @@ pub(crate) fn compress_block_with_post_split( scratch.partitions.clear(); scratch.prefix_sums.rebuild(&scratch.parts.sequences); let mut workspace = scratch.estimator_workspace.take().unwrap_or_default(); + let built_huff = core::mem::take(&mut workspace.built_huff); // Reuse the estimator's inner scratch across frames instead of // allocating a fresh `CompressedBlockScratch` (count tables + Vecs) // every block-split. Lazily created on the first split. @@ -384,12 +429,18 @@ pub(crate) fn compress_block_with_post_split( parts: &scratch.parts, prefix_sums: &scratch.prefix_sums, block_entry: ProbeEntryState { - last_huff_table: state.last_huff_table.clone(), + huff: if state.last_huff_table.is_some() { + HuffRef::BlockEntry + } else { + HuffRef::None + }, ll_previous: state.fse_tables.ll_previous.clone(), ml_previous: state.fse_tables.ml_previous.clone(), of_previous: state.fse_tables.of_previous.clone(), offset_hist: state.offset_hist, }, + entry_huff: state.last_huff_table.as_ref(), + built_huff, scratch_state: CompressState { matcher: EntropyOnlyMatcher, // The splitter's scratch state never reaches the raw-skip, which @@ -398,7 +449,9 @@ pub(crate) fn compress_block_with_post_split( // Inherited rather than re-resolved: this scratch state stands in // for the same compressor on the same CPU. copy_tier: state.copy_tier, - last_huff_table: state.last_huff_table.clone(), + // Probes read the table they repeat from `entry_huff` / + // `built_huff`; this slot stays empty. + last_huff_table: None, huff_table_spare: None, huff_rollback: None, // Lent, not created: the estimator builds a table per split @@ -406,7 +459,7 @@ pub(crate) fn compress_block_with_post_split( // on every post-split block and drop them at the end of it. Handed // back below, so the emitter that follows keeps using the same one. huff_weights: core::mem::take(&mut state.huff_weights), - fse_tables: clone_fse_tables(&state.fse_tables), + fse_tables: probe_fse_tables(&state.fse_tables), block_scratch: inner_scratch, offset_hist: state.offset_hist, strategy_tag: state.strategy_tag, @@ -419,15 +472,37 @@ pub(crate) fn compress_block_with_post_split( estimator.derive_block_splits(0, scratch.parts.sequences.len(), &mut scratch.partitions); scratch.partitions.push(scratch.parts.sequences.len()); workspace = estimator.workspace; - scratch.estimator_workspace = Some(workspace); // Stash the inner scratch back for the next frame (its buffers stay // allocated; the estimator clears them per use), and take the weight // builder's buffers back so the emitter and the next block reuse them. let CompressState { block_scratch: inner_block_scratch, - huff_weights, + mut huff_weights, + fse_tables: probe_tables, .. } = estimator.scratch_state; + // The last probe's tables join the pool for the next block's probes. + workspace.recycle_previous(probe_tables.ll_previous); + workspace.recycle_previous(probe_tables.ml_previous); + workspace.recycle_previous(probe_tables.of_previous); + for handle in [ + probe_tables.ll_next, + probe_tables.ml_next, + probe_tables.of_next, + ] + .into_iter() + .flatten() + { + workspace.recycle_fse(handle); + } + // The tables the probes built go back to the builder for its next tables, + // and their emptied arena to the workspace for the next block. + let mut built_huff = estimator.built_huff; + for table in built_huff.drain(..) { + huff_weights.recycle(table); + } + workspace.built_huff = built_huff; + scratch.estimator_workspace = Some(workspace); state.huff_weights = huff_weights; scratch.estimator_inner = Some(Box::new(inner_block_scratch)); @@ -830,15 +905,41 @@ struct EstimatorWorkspace { ll_counts: Box<[usize; 256]>, ml_counts: Box<[usize; 256]>, of_counts: Box<[usize; 256]>, - sequences: Vec, + /// FSE table handles no probe state holds any more, for the next probe to + /// build into. Every one is unique, so a build writes it in place. + spare_fse: Vec, + /// The arena a block's probes keep the Huffman tables they build in + /// ([`SplitEstimator::built_huff`]), empty between blocks: the tables go back + /// to the weight builder, the room stays for the next block's probes. + built_huff: Vec, } impl EstimatorWorkspace { - /// The four boxed count tables plus whatever the sequence buffer has grown - /// to. All four boxes are always present once the workspace exists. + /// The four boxed count tables, always present once the workspace exists, + /// the pooled tables with their reference counts, and the room kept for + /// the probes' Huffman tables. fn heap_size(&self) -> usize { 4 * core::mem::size_of::<[usize; 256]>() - + self.sequences.capacity() * core::mem::size_of::() + + self.built_huff.capacity() * core::mem::size_of::() + + self.spare_fse.capacity() * core::mem::size_of::() + + self.spare_fse.len() + * (core::mem::size_of::() + + crate::encoding::frame_compressor::shared_table_overhead()) + } + + /// Pool a handle, unless something else still holds it: a shared table + /// cannot be built into, so it is only let go. + fn recycle_fse(&mut self, mut handle: SharedFseTable) { + if SharedFseTable::get_mut(&mut handle).is_some() { + self.spare_fse.push(handle); + } + } + + /// Pool the custom tables of an axis state that is going away. + fn recycle_previous(&mut self, previous: Option) { + if let Some(PreviousFseTable::Custom(handle)) = previous { + self.recycle_fse(handle); + } } } @@ -849,7 +950,8 @@ impl Default for EstimatorWorkspace { ll_counts: Box::new([0; 256]), ml_counts: Box::new([0; 256]), of_counts: Box::new([0; 256]), - sequences: Vec::new(), + spare_fse: Vec::new(), + built_huff: Vec::new(), } } } @@ -862,36 +964,86 @@ impl Default for EstimatorWorkspace { /// FSE bit-level write. Splitter probes use this path to get the same byte /// count `encode_block_parts` would produce while saving the dominant /// `encode_sequences` write cost on every probe. +#[cfg(test)] fn estimate_block_parts_size( state: &mut CompressState, literals_vec: &[u8], raw_sequences: &[RawSequence], workspace: &mut EstimatorWorkspace, ) -> usize { - // The probe cannot fill in place: it walks sub-ranges of the block's - // sequences repeatedly, from a scratch history, while the array itself is - // borrowed immutably by the estimator for the whole search. So it keeps a - // copy — which costs nothing that matters, since block splitting only runs - // from level 11 up and never on the band this array's copy was hurting. - workspace.sequences.clear(); - if workspace.sequences.capacity() < raw_sequences.len() { - workspace - .sequences - .reserve_exact(raw_sequences.len() - workspace.sequences.len()); - } - workspace.sequences.extend_from_slice(raw_sequences); - fill_wire_offsets( - &mut workspace.sequences, - &mut state.offset_hist, - matches!( - state.strategy_tag, - crate::encoding::strategy::StrategyTag::Fast - ), + let mut sums = SequencePrefixSums::default(); + sums.rebuild(raw_sequences); + let previous = state.last_huff_table.take(); + let tables = &mut state.fse_tables; + let previous_fse = [ + tables.ll_previous.take(), + tables.ml_previous.take(), + tables.of_previous.take(), + ]; + let (bytes, outcome, decisions) = estimate_block_parts_size_with( + state, + previous.as_ref(), + [ + previous_fse[0].as_ref(), + previous_fse[1].as_ref(), + previous_fse[2].as_ref(), + ], + literals_vec, + raw_sequences, + sums.codes(0, raw_sequences.len()), + workspace, ); + state.last_huff_table = match outcome { + HuffOutcome::Keep => previous, + HuffOutcome::Clear => None, + HuffOutcome::New(table) => Some(table), + }; + let [ll, ml, of] = previous_fse; + let tables = &mut state.fse_tables; + tables.ll_previous = ll; + tables.ml_previous = ml; + tables.of_previous = of; + remember_last_used_tables(tables, decisions); + for next in [ + &mut tables.ll_next, + &mut tables.ml_next, + &mut tables.of_next, + ] { + if let Some(handle) = next.take() { + workspace.recycle_fse(handle); + } + } + bytes +} + +/// What pricing a literals section did to the Huffman table the next section +/// may repeat: keep the one it was given, drop it, or take a new one. A probe +/// reads the previous table and reports this, rather than rewriting a copy of +/// it, so the table it starts from is borrowed. +enum HuffOutcome { + Keep, + Clear, + New(huff0_encoder::HuffmanTable), +} - let lit_bytes = estimate_literals_section_bytes( +/// `estimate_block_parts_size` for sequences whose length codes are already +/// derived: the splitter's probes, which price many ranges of one block. The +/// tables the section may repeat are borrowed, the Huffman table as `previous` +/// and the FSE tables as `previous_fse`, and what the section does with each is +/// returned beside the size, a built FSE table in its `*_next` slot of +/// `state.fse_tables`; the offset history is advanced in `state`. +fn estimate_block_parts_size_with( + state: &mut CompressState, + previous: Option<&huff0_encoder::HuffmanTable>, + previous_fse: [Option<&PreviousFseTable>; 3], + literals_vec: &[u8], + raw_sequences: &[RawSequence], + codes: LengthCodes<'_>, + workspace: &mut EstimatorWorkspace, +) -> (usize, HuffOutcome, [LastUsedTable; 3]) { + let (lit_bytes, outcome) = estimate_literals_section_bytes( literals_vec, - &mut state.last_huff_table, + previous, &mut workspace.lit_counts, state.strategy_tag, state.huf_optimal_search, @@ -900,20 +1052,22 @@ fn estimate_block_parts_size( literals_suspected_incompressible(literals_vec.len(), raw_sequences.len()), ); - let seq_bytes = if workspace.sequences.is_empty() { - 1 + // A section without sequences writes no tables, so every axis keeps its own. + let (seq_bytes, decisions) = if raw_sequences.is_empty() { + (1, [LastUsedTable::Keep; 3]) } else { estimate_sequences_section_bytes( - &workspace.sequences, + raw_sequences, + codes, + &mut state.offset_hist, + previous_fse, &mut state.fse_tables, - &mut workspace.ll_counts, - &mut workspace.ml_counts, - &mut workspace.of_counts, + workspace, state.strategy_tag, ) }; - lit_bytes + seq_bytes + (lit_bytes + seq_bytes, outcome, decisions) } // One argument over the lint's threshold. Every one of them is a distinct @@ -923,20 +1077,25 @@ fn estimate_block_parts_size( #[allow(clippy::too_many_arguments)] fn estimate_literals_section_bytes( literals: &[u8], - last_huff: &mut Option, + last_huff: Option<&huff0_encoder::HuffmanTable>, counts: &mut [usize; 256], strategy: crate::encoding::strategy::StrategyTag, huf_search: bool, lit_disabled: bool, weight_scratch: &mut huff0_encoder::WeightScratch, suspected_incompressible: bool, -) -> usize { +) -> (usize, HuffOutcome) { + let raw = || { + ( + uncompressed_literals_header_bytes(literals.len()) + literals.len(), + HuffOutcome::Clear, + ) + }; // Mirror `encode_block_parts` literal-mode branches // **in the same order**. The disabled gate (negative levels: raw literals, // no Huffman) is checked FIRST exactly as the emitter does. if lit_disabled { - *last_huff = None; - return uncompressed_literals_header_bytes(literals.len()) + literals.len(); + return raw(); } // The emitter pre-checks `all_identical` // (any non-empty section) BEFORE the `min_lits` gate — RLE and raw @@ -947,13 +1106,14 @@ fn estimate_literals_section_bytes( // regardless of strategy. Estimator must use the same ordering and // predicate so probe costs match emit byte-for-byte. if !literals.is_empty() && all_bytes_identical(literals) { - *last_huff = None; - return uncompressed_literals_header_bytes(literals.len()) + 1; + return ( + uncompressed_literals_header_bytes(literals.len()) + 1, + HuffOutcome::Clear, + ); } let min_lits = min_literals_to_compress(strategy, last_huff.is_some()); if literals.len() < min_lits { - *last_huff = None; - return uncompressed_literals_header_bytes(literals.len()) + literals.len(); + return raw(); } // Upstream zstd preferRepeat fast-path: skip the histogram + @@ -971,18 +1131,16 @@ fn estimate_literals_section_bytes( // short-circuit so we still fall through to rebuild when the // prior table can't encode the current literals. if prefer_repeat_eligible(strategy, literals.len()) - && let Some(prev) = last_huff.as_ref() + && let Some(prev) = last_huff && let Some(reuse_payload) = estimate_huff_payload_bytes_checked(prev, literals) { let compressed_header = compressed_literals_header_bytes(literals.len()); let total = compressed_header + reuse_payload; // no tree_desc on reuse - let raw_section_bytes = uncompressed_literals_header_bytes(literals.len()) + literals.len(); let huf_section_size = total - compressed_header; if use_raw_literal_fallback(huf_section_size, literals.len(), strategy) { - *last_huff = None; - return raw_section_bytes; + return raw(); } - return total; + return (total, HuffOutcome::Keep); } // Mirror the emitter's end-sample shortcut, in the same position. Without @@ -990,8 +1148,7 @@ fn estimate_literals_section_bytes( // Huffman-compressed here and is emitted raw there, and the splitter picks // a partition on a price the emitter cannot produce. if suspected_incompressible && end_samples_look_flat(literals, counts) { - *last_huff = None; - return uncompressed_literals_header_bytes(literals.len()) + literals.len(); + return raw(); } let (max_sym, largest_count) = crate::histogram::count_bytes(literals, counts); @@ -999,8 +1156,7 @@ fn estimate_literals_section_bytes( // byte-for-byte (flat histogram → raw section, no tree build) so // splitter probe costs match what the emitter writes. if largest_count <= (literals.len() >> 7) + 4 { - *last_huff = None; - return uncompressed_literals_header_bytes(literals.len()) + literals.len(); + return raw(); } // Mutable because the size query is what encodes the weight description // into the table's own buffer, so the emitter that follows reads it. @@ -1010,12 +1166,13 @@ fn estimate_literals_section_bytes( weight_scratch, ); - let Some(new_desc) = new_table.writeable_table_description_size() else { - *last_huff = None; + let Some(new_desc) = + new_table.writeable_table_description_size(weight_scratch.weight_fse_table()) + else { // Nothing downstream reads this table; hand its buffers to the next // build rather than dropping them. weight_scratch.recycle(new_table); - return uncompressed_literals_header_bytes(literals.len()) + literals.len(); + return raw(); }; // For lit_size ≥ 256, upstream zstd `compress_literals` calls `encoder.encode4x` // which splits the data in 4 streams with a 6-byte jumptable and per-stream @@ -1032,20 +1189,12 @@ fn estimate_literals_section_bytes( // Using the 4-stream `estimate_huff_payload_bytes_checked` here would // disagree with the encoder and bias the splitter to pick a different // table than the encoder ultimately emits. - let use_new = decide_huff_reuse_like_encoder( - &new_table, - last_huff.as_ref(), - new_desc, - literals, - counts, - strategy, - ); + let use_new = + decide_huff_reuse_like_encoder(&new_table, last_huff, new_desc, literals, counts, strategy); let reuse_payload = if !use_new { // Safe to recompute with 4-stream model now that the table is chosen: // the chosen-table path always returns the actual wire cost. - last_huff - .as_ref() - .and_then(|t| estimate_huff_payload_bytes_checked(t, literals)) + last_huff.and_then(|t| estimate_huff_payload_bytes_checked(t, literals)) } else { None }; @@ -1075,86 +1224,132 @@ fn estimate_literals_section_bytes( let raw_section_bytes = uncompressed_literals_header_bytes(literals.len()) + literals.len(); let huf_section_size = total - compressed_header; // tree_desc + payload, no lhSize if use_raw_literal_fallback(huf_section_size, literals.len(), strategy) { - *last_huff = None; weight_scratch.recycle(new_table); - return raw_section_bytes; + return (raw_section_bytes, HuffOutcome::Clear); } if use_new { - // The table this displaces is the one to recycle; the new one is kept. - if let Some(displaced) = last_huff.replace(new_table) { - weight_scratch.recycle(displaced); - } + (total, HuffOutcome::New(new_table)) } else { weight_scratch.recycle(new_table); + (total, HuffOutcome::Keep) } - total } +/// Price a sequence section. The offset codes are the one part that depends on +/// the history the section starts from, so they are resolved here, in one pass +/// that advances `offset_hist` as the emitter would; the length codes and +/// their extra bits come precomputed in `codes`. Each histogram is built once +/// and handed to the table selection, which the emitter's path counts again. +/// +/// The tables the section may repeat are `previous` (LL, ML, OF), borrowed: a +/// probe reads them and does not take a handle. What the section decided for +/// each axis is returned beside the size, a built table left in that axis's +/// `*_next` slot of `fse_tables`, for the caller to commit. fn estimate_sequences_section_bytes( sequences: &[RawSequence], + codes: LengthCodes<'_>, + offset_hist: &mut [u32; 3], + previous: [Option<&PreviousFseTable>; 3], fse_tables: &mut FseTables, - ll_counts: &mut [usize; 256], - ml_counts: &mut [usize; 256], - of_counts: &mut [usize; 256], + workspace: &mut EstimatorWorkspace, strategy: crate::encoding::strategy::StrategyTag, -) -> usize { - ll_counts.fill(0); - ml_counts.fill(0); +) -> (usize, [LastUsedTable; 3]) { + let EstimatorWorkspace { + ll_counts, + ml_counts, + of_counts, + .. + } = workspace; + let (ll_counts, ml_counts, of_counts) = (&mut **ll_counts, &mut **ml_counts, &mut **of_counts); + debug_assert_eq!(codes.ll.len(), sequences.len()); + let histogram = |codes: &[u8], counts: &mut [usize; 256]| { + counts.fill(0); + let mut max = 0u8; + for &code in codes { + counts[code as usize] += 1; + max = max.max(code); + } + max as usize + }; + let ll_max = histogram(codes.ll, ll_counts); + let ml_max = histogram(codes.ml, ml_counts); of_counts.fill(0); - let mut extra_bits: usize = 0; - for seq in sequences { - let (ll, _, ll_bits) = encode_literal_length(seq.ll); - let (ml, _, ml_bits) = encode_match_len(seq.ml); - let (of, _, _) = encode_offset(seq.off_base); - ll_counts[ll as usize] += 1; - ml_counts[ml as usize] += 1; + let mut of_max = 0usize; + let mut of_bits = 0usize; + let mut hist = *offset_hist; + let mut count_offset = |off_base: u32| { + let (of, _, _) = encode_offset(off_base); of_counts[of as usize] += 1; + of_max = of_max.max(of as usize); // Upstream zstd: OF code's value equals its additional-bits width. - extra_bits += ll_bits + ml_bits + of as usize; + of_bits += of as usize; + }; + if matches!(strategy, crate::encoding::strategy::StrategyTag::Fast) { + for seq in sequences { + count_offset(encode_offset_with_history_fast( + seq.off_base, + seq.ll, + &mut hist, + )); + } + } else { + for seq in sequences { + count_offset(encode_offset_with_history(seq.off_base, seq.ll, &mut hist)); + } } + *offset_hist = hist; + let extra_bits = codes.bits + of_bits; + let total = sequences.len(); // Destructured for the same reason as the emitter: the default accessors // borrow the whole struct, which would collide with the `*_next` slots. let FseTables { - ll_previous, - ml_previous, - of_previous, ll_next, ml_next, of_next, ll_default, ml_default, of_default, + .. } = fse_tables; let ll_default: &FSETable = ll_default; let ml_default: &FSETable = ml_default; let of_default: &FSETable = of_default; + let [ll_previous, ml_previous, of_previous] = previous; - // Same `choose_table` calls as the real encoder — counts the iterator - // internally, identical decision path. - let ll_mode = choose_table( - ll_previous.as_ref(), + // The table selection the real encoder makes, from the same histograms. + let ll_mode = choose_table_from_counts( + ll_previous, ll_default, - sequences.iter().map(|seq| encode_literal_length(seq.ll).0), + ll_counts, + total, + ll_max, 9, strategy, + None, ll_next, ); - let ml_mode = choose_table( - ml_previous.as_ref(), + let ml_mode = choose_table_from_counts( + ml_previous, ml_default, - sequences.iter().map(|seq| encode_match_len(seq.ml).0), + ml_counts, + total, + ml_max, 9, strategy, + None, ml_next, ); - let of_mode = choose_table( - of_previous.as_ref(), + let of_mode = choose_table_from_counts( + of_previous, of_default, - sequences.iter().map(|seq| encode_offset(seq.off_base).0), + of_counts, + total, + of_max, 8, strategy, + None, of_next, ); @@ -1184,28 +1379,22 @@ fn estimate_sequences_section_bytes( }; let stream_bytes = (bit_content + padding_bits) / 8; - // Mirror state mutation done by `encode_block_parts`. + // What `encode_block_parts` would commit, left to the caller to commit. let decisions = [ into_last_used_table(ll_mode), into_last_used_table(ml_mode), into_last_used_table(of_mode), ]; - remember_last_used_tables(fse_tables, decisions); - // The emitter keeps the handle a commit displaces, to build the next - // block's table into. A probe must not: the splitter holds many of these - // states at once, and a spare per axis per probe doubles the tables alive - // at any moment. Dropping it leaves a probe with exactly what it needs, - // which is what the allocate-per-table form gave it. - fse_tables.ll_next = None; - fse_tables.ml_next = None; - fse_tables.of_next = None; - - nb_seq_header - + mode_byte - + ll_table_desc_bytes - + of_table_desc_bytes - + ml_table_desc_bytes - + stream_bytes + + ( + nb_seq_header + + mode_byte + + ll_table_desc_bytes + + of_table_desc_bytes + + ml_table_desc_bytes + + stream_bytes, + decisions, + ) } /// Bit cost of a sequence section under `mode`, matching what @@ -1687,28 +1876,10 @@ fn highest_used_code(counts: &[usize; 256]) -> usize { .unwrap_or(0) } -/// [`fill_and_count`] without the histogram, for the block-split estimator: it -/// prices sub-ranges repeatedly from a scratch history and counts them itself. -fn fill_wire_offsets( - raw_sequences: &mut [RawSequence], - offset_hist: &mut [u32; 3], - fast_repcode: bool, -) { - // Local copy for the same reason as `fill_and_count`. - let mut hist = *offset_hist; - if fast_repcode { - for seq in raw_sequences.iter_mut() { - seq.off_base = encode_offset_with_history_fast(seq.off_base, seq.ll, &mut hist); - } - } else { - for seq in raw_sequences.iter_mut() { - seq.off_base = encode_offset_with_history(seq.off_base, seq.ll, &mut hist); - } - } - *offset_hist = hist; -} - -fn clone_fse_tables(fse_tables: &FseTables) -> FseTables { +/// The FSE tables a split probe's scratch state works with: the defaults, and +/// no previous table, since every probe reads the one it may repeat from the +/// state it starts from. +fn probe_fse_tables(fse_tables: &FseTables) -> FseTables { // The `*_default` fields are cfg-typed via the // [`crate::fse::fse_encoder::FseDefaultTable`] alias — // `&'static FSETable` on atomic / `critical-section` targets @@ -1730,17 +1901,17 @@ fn clone_fse_tables(fse_tables: &FseTables) -> FseTables { ll_default: fse_tables.ll_default, #[cfg(not(any(target_has_atomic = "ptr", feature = "critical-section")))] ll_default: fse_tables.ll_default.clone(), - ll_previous: fse_tables.ll_previous.clone(), + ll_previous: None, #[cfg(any(target_has_atomic = "ptr", feature = "critical-section"))] ml_default: fse_tables.ml_default, #[cfg(not(any(target_has_atomic = "ptr", feature = "critical-section")))] ml_default: fse_tables.ml_default.clone(), - ml_previous: fse_tables.ml_previous.clone(), + ml_previous: None, #[cfg(any(target_has_atomic = "ptr", feature = "critical-section"))] of_default: fse_tables.of_default, #[cfg(not(any(target_has_atomic = "ptr", feature = "critical-section")))] of_default: fse_tables.of_default.clone(), - of_previous: fse_tables.of_previous.clone(), + of_previous: None, // Empty, not blank tables: a probe gets its own slots so it cannot // overwrite what the emitter is describing, but most probes never // build a custom table and must not pay for one. @@ -1757,17 +1928,31 @@ fn clone_fse_tables(fse_tables: &FseTables) -> FseTables { /// estimator replaces. #[derive(Clone)] struct ProbeEntryState { - last_huff_table: Option, + huff: HuffRef, ll_previous: Option, ml_previous: Option, of_previous: Option, offset_hist: [u32; 3], } +/// Which Huffman table a probe state may repeat. Tables are never copied into +/// a state: the block's entry table is borrowed from the compressor, and a +/// table a probe builds is kept once in the estimator's arena. +#[derive(Clone, Copy)] +enum HuffRef { + None, + BlockEntry, + Built(usize), +} + struct SplitEstimator<'a> { parts: &'a EncodedBlockParts, prefix_sums: &'a SequencePrefixSums, block_entry: ProbeEntryState, + /// The table the block starts from, borrowed. + entry_huff: Option<&'a huff0_encoder::HuffmanTable>, + /// Every table a probe built, addressed by [`HuffRef::Built`]. + built_huff: Vec, scratch_state: CompressState, workspace: EstimatorWorkspace, } @@ -1794,15 +1979,37 @@ impl SplitEstimator<'_> { } else { lit_start + lit_len }; - self.scratch_state.last_huff_table = entry.last_huff_table.clone(); - self.scratch_state.fse_tables.ll_previous = entry.ll_previous.clone(); - self.scratch_state.fse_tables.ml_previous = entry.ml_previous.clone(); - self.scratch_state.fse_tables.of_previous = entry.of_previous.clone(); + // Every table the probe may repeat is read where the entry state keeps + // it; a handle is taken only for what the post state keeps. A build slot + // comes from the pool, so a probe that builds a table writes into one it + // already has. + let tables = &mut self.scratch_state.fse_tables; + for next in [ + &mut tables.ll_next, + &mut tables.ml_next, + &mut tables.of_next, + ] { + if next.is_none() { + *next = self.workspace.spare_fse.pop(); + } + } self.scratch_state.offset_hist = entry.offset_hist; - let emitted_payload = estimate_block_parts_size( + let previous = match entry.huff { + HuffRef::None => None, + HuffRef::BlockEntry => self.entry_huff, + HuffRef::Built(at) => Some(&self.built_huff[at]), + }; + let (emitted_payload, outcome, decisions) = estimate_block_parts_size_with( &mut self.scratch_state, + previous, + [ + entry.ll_previous.as_ref(), + entry.ml_previous.as_ref(), + entry.of_previous.as_ref(), + ], &self.parts.literals[lit_start..lit_end], &self.parts.sequences[start_idx..end_idx], + self.prefix_sums.codes(start_idx, end_idx), &mut self.workspace, ); let source_len = (lit_end - lit_start) + match_len; @@ -1814,21 +2021,42 @@ impl SplitEstimator<'_> { emitted_payload } + 3; // Real emit on raw fallback restores the entry state — see - // `emit_single_sequence_block`'s saved-state restore branch. + // `emit_single_sequence_block`'s saved-state restore branch. A table the + // probe built stays in its slot for the next probe to build into. let post = if raw_fallback { + if let HuffOutcome::New(table) = outcome { + self.scratch_state.huff_weights.recycle(table); + } entry.clone() } else { + let huff = match outcome { + HuffOutcome::Keep => entry.huff, + HuffOutcome::Clear => HuffRef::None, + HuffOutcome::New(table) => { + self.built_huff.push(table); + HuffRef::Built(self.built_huff.len() - 1) + } + }; + let tables = &mut self.scratch_state.fse_tables; + let [ll, ml, of] = decisions; ProbeEntryState { - last_huff_table: self.scratch_state.last_huff_table.clone(), - ll_previous: self.scratch_state.fse_tables.ll_previous.clone(), - ml_previous: self.scratch_state.fse_tables.ml_previous.clone(), - of_previous: self.scratch_state.fse_tables.of_previous.clone(), + huff, + ll_previous: decided_previous(ll, &entry.ll_previous, &mut tables.ll_next), + ml_previous: decided_previous(ml, &entry.ml_previous, &mut tables.ml_next), + of_previous: decided_previous(of, &entry.of_previous, &mut tables.of_next), offset_hist: self.scratch_state.offset_hist, } }; (cost, raw_fallback, post) } + /// Pool the tables of a probe state nothing will read again. + fn recycle_state(&mut self, state: ProbeEntryState) { + self.workspace.recycle_previous(state.ll_previous); + self.workspace.recycle_previous(state.ml_previous); + self.workspace.recycle_previous(state.of_previous); + } + fn derive_block_splits( &mut self, start_idx: usize, @@ -1841,7 +2069,9 @@ impl SplitEstimator<'_> { return; } let entry = self.block_entry.clone(); - let (full, full_raw_fallback, _) = self.estimate_subblock_size(start_idx, end_idx, &entry); + let (full, full_raw_fallback, full_post) = + self.estimate_subblock_size(start_idx, end_idx, &entry); + self.recycle_state(full_post); // G3 — whole-block bail-out before partition split. Upstream zstd // `ZSTD_compressSubBlock_multi` (`zstd_compress_superblock.c:530-532`) // bails when `estBlockSize > srcSize` (strict). Our trigger is @@ -1887,7 +2117,8 @@ impl SplitEstimator<'_> { if full_raw_fallback { return; } - self.derive_block_splits_with_full(start_idx, end_idx, full, entry, partitions); + let exit = self.derive_block_splits_with_full(start_idx, end_idx, full, entry, partitions); + self.recycle_state(exit); } /// Returns the post-emit state at `end_idx` produced by whichever @@ -1910,6 +2141,7 @@ impl SplitEstimator<'_> { // exit state is the post-state of that single-partition probe. let (_cost, _raw_fallback, post) = self.estimate_subblock_size(start_idx, end_idx, &entry); + self.recycle_state(entry); return post; } let mid_idx = (start_idx + end_idx) / 2; @@ -1918,7 +2150,9 @@ impl SplitEstimator<'_> { // not from the parent's block-entry state. Without this propagation // `second` is evaluated as a fresh-block start, biasing the // `first + second < full` decision toward overly optimistic splits. - let (second, _, _) = self.estimate_subblock_size(mid_idx, end_idx, &first_post); + let (second, _, second_post) = self.estimate_subblock_size(mid_idx, end_idx, &first_post); + self.recycle_state(second_post); + self.recycle_state(first_post); if first + second < full { // If the left side gets further split, the true state at // `mid_idx` is the left subtree's exit state, not `first_post`. @@ -1935,6 +2169,7 @@ impl SplitEstimator<'_> { } // No split here — this range will be emitted as one partition. let (_cost, _raw_fallback, post) = self.estimate_subblock_size(start_idx, end_idx, &entry); + self.recycle_state(entry); post } } @@ -2045,47 +2280,8 @@ fn fse_bit_cost(counts: &[usize; 256], max_symbol: usize, table: &FSETable) -> O Some(cost >> 8) } -fn choose_table<'a>( - previous: Option<&'a PreviousFseTable>, - default_table: &'a FSETable, - data: impl Iterator, - max_log: u8, - strategy: crate::encoding::strategy::StrategyTag, - next: &'a mut Option, -) -> FseTableMode<'a> { - // Collect symbol distribution, tracking the highest code so the selector - // skips the full-256 reverse scan (see `choose_table_from_counts`). - let mut counts = [0usize; 256]; - let mut total = 0usize; - let mut max_symbol = 0usize; - for symbol in data { - let symbol = symbol as usize; - counts[symbol] += 1; - total += 1; - max_symbol = max_symbol.max(symbol); - } - choose_table_from_counts( - previous, - default_table, - &mut counts, - total, - max_symbol, - max_log, - strategy, - // Estimator-only path (no emitted table): price the unadjusted histogram, - // matching upstream's `ZSTD_NCountCost`. - None, - next, - ) -} - -/// Same decision logic as [`choose_table`] but takes pre-computed -/// symbol counts and total directly. Hot-path callers in -/// `compress_literals_and_sequences` use this overload to avoid -/// re-iterating the sequence vec three times (one pass per -/// ll/ml/of stream); the iterator form is kept for the cost -/// estimator's call sites where the data is already in iterator -/// form. +/// Choose an FSE table mode from a stream's symbol counts, which every caller +/// has already built while visiting the codes once. // The eight inputs are the cohesive FSE-table-selection set, each carrying its // own perf / correctness rationale below (the `&mut` histogram for the no-copy // emit build, the caller-tracked `max_symbol` / `last_code` that avoid a @@ -2367,6 +2563,26 @@ enum LastUsedTable { Encoded, } +/// The table an axis repeats after a probe decided `decision` from `entry`: +/// what [`commit_last_used_table`] would leave in the previous slot, taken for +/// a probe state. Only a repeat takes a handle to the entry's table; a built +/// table moves out of `next`, which the next probe refills from the pool. +fn decided_previous( + decision: LastUsedTable, + entry: &Option, + next: &mut Option, +) -> Option { + match decision { + LastUsedTable::Keep => entry.clone(), + LastUsedTable::Default => Some(PreviousFseTable::Default), + LastUsedTable::Rle(symbol) => Some(PreviousFseTable::Rle(symbol)), + LastUsedTable::Encoded => Some(PreviousFseTable::Custom( + next.take() + .expect("an encoded axis built its table into the slot"), + )), + } +} + fn into_last_used_table(mode: FseTableMode<'_>) -> LastUsedTable { match mode { FseTableMode::Encoded(_) => LastUsedTable::Encoded, @@ -3096,7 +3312,8 @@ fn compress_literals( weight_scratch, ); - let Some(new_table_description_size) = new_encoder_table.writeable_table_description_size() + let Some(new_table_description_size) = + new_encoder_table.writeable_table_description_size(weight_scratch.weight_fse_table()) else { raw_literals(literals, writer); weight_scratch.recycle(new_encoder_table); diff --git a/zstd/src/encoding/blocks/compressed/tests.rs b/zstd/src/encoding/blocks/compressed/tests.rs index a09228bdf..0715512bd 100644 --- a/zstd/src/encoding/blocks/compressed/tests.rs +++ b/zstd/src/encoding/blocks/compressed/tests.rs @@ -1,14 +1,48 @@ use super::{ - FseTableMode, LastUsedTable, RawSequence, choose_table, emit_single_sequence_block, + FseTableMode, LastUsedTable, RawSequence, choose_table_from_counts, emit_single_sequence_block, encode_literal_length, encode_match_len, encode_offset_with_history, min_gain, min_literals_to_compress, previous_table, remember_last_used_tables, }; -use crate::encoding::frame_compressor::{CompressState, FseTables, PreviousFseTable}; +use crate::encoding::frame_compressor::{ + CompressState, FseTables, PreviousFseTable, SharedFseTable, +}; use crate::encoding::strategy::StrategyTag; -use crate::fse::fse_encoder::build_table_from_symbol_counts; +use crate::fse::fse_encoder::{FSETable, build_table_from_symbol_counts}; use crate::huff0::huff0_encoder; use alloc::vec::Vec; +/// The table selection for a stream given as codes: counts them, then selects +/// as the encoder does, pricing the unadjusted histogram. +fn choose_table<'a>( + previous: Option<&'a PreviousFseTable>, + default_table: &'a FSETable, + data: impl Iterator, + max_log: u8, + strategy: StrategyTag, + next: &'a mut Option, +) -> FseTableMode<'a> { + let mut counts = [0usize; 256]; + let mut total = 0usize; + let mut max_symbol = 0usize; + for symbol in data { + let symbol = symbol as usize; + counts[symbol] += 1; + total += 1; + max_symbol = max_symbol.max(symbol); + } + choose_table_from_counts( + previous, + default_table, + &mut counts, + total, + max_symbol, + max_log, + strategy, + None, + next, + ) +} + fn tables_match( lhs: &crate::fse::fse_encoder::FSETable, rhs: &crate::fse::fse_encoder::FSETable, @@ -198,7 +232,7 @@ fn decide_huff_reuse_prefer_repeat_forces_reuse_for_fast_band() { skewed_literals.extend((0..16u8).map(|i| 200 + i)); let mut new_tbl = huff0_encoder::HuffmanTable::build_from_data(&skewed_literals); let new_desc = new_tbl - .writeable_table_description_size() + .writeable_table_description_size(&mut crate::fse::fse_encoder::FSETable::blank()) .expect("non-empty table emits a description"); // The decision reads its sizes off the histogram of the very literals it diff --git a/zstd/src/encoding/bt/mod.rs b/zstd/src/encoding/bt/mod.rs index 6f0aa0a31..bd218287a 100644 --- a/zstd/src/encoding/bt/mod.rs +++ b/zstd/src/encoding/bt/mod.rs @@ -4,7 +4,7 @@ //! model (`opt_state`), the optimal-parser scratch buffers //! (`opt_*_scratch` / `opt_*_generation` / `opt_*_stamp`), and the //! LDM long-distance match buffer (`ldm_sequences`). Method bodies -//! (BT walk, `bt_insert_step_no_rebase`, `bt_update_tree_until`, +//! (BT walk, `bt_insert_range`, `bt_update_tree_until`, //! `build_optimal_plan*`, `collect_optimal_candidates*`, //! `emit_optimal_plan`, …) still live on `HcMatchGenerator` and will //! move onto `impl BtMatcher` once Stage 3b threads @@ -155,6 +155,21 @@ impl BtMatcher { false } + /// Append the long-distance candidate after the search when it is at + /// least `min_match_len` and longer than every candidate the search found + /// (upstream zstd `ZSTD_optLdm_maybeAddMatch`, zstd_opt.c). The search + /// ladder keeps `out` sorted by strictly increasing length, so its last + /// entry is the bar. + #[inline(always)] + pub(crate) fn push_ldm_candidate( + out: &mut Vec, + ldm: MatchCandidate, + min_match_len: usize, + ) { + let mut best_len = out.last().map_or(0, |c| c.match_len); + let _ = Self::push_candidate_ladder(out, &mut best_len, ldm, min_match_len); + } + pub(crate) fn new() -> Self { Self { opt_state: HcOptState::new(), @@ -379,6 +394,7 @@ impl BtMatcher { store, price_arena, candidates_searched_at: _, + pass: _, } = buffers; candidates.clear(); self.opt_nodes_scratch = nodes; @@ -572,47 +588,41 @@ impl BtMatcher { } } - /// Upstream zstd parity: replay an already-emitted plan segment through the - /// `optStatePtr_t` stats updater so the next parse pass sees frozen - /// counts. Pure static helper — only mutates the caller-owned - /// `opt_state` / `reps` / `literals_start`. - pub(crate) fn update_plan_stats_segment( - current: &[u8], - current_len: usize, - plan: &[HcOptimalSequence], + /// Upstream zstd parity: `ZSTD_updateStats` for one sequence the parser + /// has just settled, called from its traceback so the next segment prices + /// against counts that include it. `literals_start` is the block offset + /// the previous sequence ended at and `reps` the history before this one; + /// both advance. The caller refreshes the base prices once the segment's + /// sequences are in. + #[inline] + pub(crate) fn record_sequence_stats( + block: &[u8], + sequence: HcOptimalSequence, literals_start: &mut usize, reps: &mut [u32; 3], opt_state: &mut HcOptState, - accurate: bool, ) { - if plan.is_empty() { + let lit_len = sequence.lit_len as usize; + let match_len = sequence.match_len as usize; + // `checked_add` on both edges so a malformed sequence can't overflow + // `usize` arithmetic before the bounds guard fires. `saturating_add` + // would have masked overflow as "clamp to usize::MAX" which then + // bypasses the `> block.len()` check. + let Some(start) = literals_start.checked_add(lit_len) else { + return; + }; + let Some(end) = start.checked_add(match_len) else { + return; + }; + if end > block.len() { return; } - for item in plan { - let lit_len = item.lit_len as usize; - let match_len = item.match_len as usize; - // `checked_add` on both edges so a malformed / partially-built - // plan can't overflow `usize` arithmetic before the - // bounds guard fires. `saturating_add` would have masked - // overflow as "clamp to usize::MAX" which then bypasses the - // `> current_len` check. - let Some(start) = literals_start.checked_add(lit_len) else { - continue; - }; - let Some(end) = start.checked_add(match_len) else { - continue; - }; - if end > current_len { - continue; - } - let literals = ¤t[*literals_start..start]; - let (off_base, next_reps) = - Self::encode_offset_with_reps(item.offset, literals.len(), *reps); - opt_state.update_stats(literals.len(), literals, off_base, match_len); - *reps = next_reps; - *literals_start = end; - } - opt_state.set_base_prices(accurate); + let literals = &block[*literals_start..start]; + let (off_base, next_reps) = + Self::encode_offset_with_reps(sequence.offset, literals.len(), *reps); + opt_state.update_stats(literals.len(), literals, off_base, match_len); + *reps = next_reps; + *literals_start = end; } /// Brings cells `start..=end` into the frontier as unreached: price `MAX`. @@ -649,7 +659,7 @@ impl BtMatcher { } #[inline(always)] - pub(crate) fn cached_literal_price( + pub(crate) fn cached_literal_price( profile: HcOptimalCostProfile, stats: &HcOptState, byte: u8, @@ -657,6 +667,12 @@ impl BtMatcher { generations: &mut [u32; HC_MAX_LIT + 1], stamp: u32, ) -> u32 { + // The integer weight is one bit scan on a frequency, no dearer than the + // probe that would stand in for it, so it is computed in place, as + // upstream does at `optLevel` 0. Only the fractional weight is cached. + if !ACCURATE { + return profile.literal_price::(stats, byte); + } // SAFETY: `byte as usize` is `0..256` and the fixed-size arrays are // `[u32; HC_MAX_LIT + 1 = 257]`, so the index is statically in bounds. // Each cached_*_price call sits inside the optimal parser per-byte @@ -666,7 +682,7 @@ impl BtMatcher { if *generations.get_unchecked(idx) == stamp { return *prices.get_unchecked(idx); } - let price = profile.literal_price(stats, byte); + let price = profile.literal_price::(stats, byte); *prices.get_unchecked_mut(idx) = price; *generations.get_unchecked_mut(idx) = stamp; price @@ -674,15 +690,16 @@ impl BtMatcher { } #[inline(always)] - pub(crate) fn cached_lit_length_price( + pub(crate) fn cached_lit_length_price( profile: HcOptimalCostProfile, stats: &HcOptState, lit_len: usize, cache: &mut [[u32; 2]], stamp: u32, ) -> u32 { - if lit_len >= cache.len() { - return profile.lit_length_price(stats, lit_len); + // Computed in place under the integer weight; see `cached_literal_price`. + if !ACCURATE || lit_len >= cache.len() { + return profile.lit_length_price::(stats, lit_len); } // SAFETY: the early-return above proves `lit_len < cache.len()`. // Each cell pairs `[price, generation]`, so the stamp check and the @@ -693,7 +710,7 @@ impl BtMatcher { if cell[1] == stamp { return cell[0]; } - let price = profile.lit_length_price(stats, lit_len); + let price = profile.lit_length_price::(stats, lit_len); cell[0] = price; cell[1] = stamp; price @@ -701,7 +718,7 @@ impl BtMatcher { } #[inline(always)] - pub(crate) fn cached_lit_length_delta_price( + pub(crate) fn cached_lit_length_delta_price( profile: HcOptimalCostProfile, stats: &HcOptState, lit_len: usize, @@ -715,21 +732,24 @@ impl BtMatcher { // No need to compute `0_usize - 1`. return 0; } - let price = Self::cached_lit_length_price(profile, stats, lit_len, cache, stamp); - let previous = Self::cached_lit_length_price(profile, stats, lit_len - 1, cache, stamp); + let price = + Self::cached_lit_length_price::(profile, stats, lit_len, cache, stamp); + let previous = + Self::cached_lit_length_price::(profile, stats, lit_len - 1, cache, stamp); price as i32 - previous as i32 } #[inline(always)] - pub(crate) fn cached_match_length_price( + pub(crate) fn cached_match_length_price( profile: HcOptimalCostProfile, stats: &HcOptState, match_len: usize, cache: &mut [[u32; 2]], stamp: u32, ) -> u32 { - if match_len >= cache.len() { - return profile.match_length_price(stats, match_len); + // Computed in place under the integer weight; see `cached_literal_price`. + if !ACCURATE || match_len >= cache.len() { + return profile.match_length_price::(stats, match_len); } // SAFETY: see `cached_lit_length_price` — paired `[price, generation]` // cells, one cache line per probe; early return proves @@ -739,7 +759,7 @@ impl BtMatcher { if cell[1] == stamp { return cell[0]; } - let price = profile.match_length_price(stats, match_len); + let price = profile.match_length_price::(stats, match_len); cell[0] = price; cell[1] = stamp; price diff --git a/zstd/src/encoding/cost_model/mod.rs b/zstd/src/encoding/cost_model/mod.rs index bb8b5ece1..bb9d01947 100644 --- a/zstd/src/encoding/cost_model/mod.rs +++ b/zstd/src/encoding/cost_model/mod.rs @@ -301,7 +301,9 @@ impl HcOptState { } } - pub(crate) fn rescale_freqs(&mut self, src: &[u8], profile: HcOptimalCostProfile) { + /// `accurate` is the strategy's price mode, which the base prices are + /// derived in. + pub(crate) fn rescale_freqs(&mut self, src: &[u8], accurate: bool) { self.price_type = HcOptPriceType::Dynamic; if self.lit_length_sum == 0 { if src.len() <= HC_PREDEF_THRESHOLD { @@ -404,7 +406,7 @@ impl HcOptState { self.match_length_sum = Self::scale_stats(&mut self.match_length_freq, 11); self.off_code_sum = Self::scale_stats(&mut self.off_code_freq, 11); } - self.set_base_prices(profile.accurate); + self.set_base_prices(accurate); } pub(crate) fn update_stats( @@ -440,9 +442,7 @@ impl HcOptState { #[derive(Copy, Clone)] pub(crate) struct HcOptimalCostProfile { - pub(crate) max_chain_depth: usize, pub(crate) sufficient_match_len: usize, - pub(crate) accurate: bool, pub(crate) favor_small_offsets: bool, } @@ -454,10 +454,10 @@ impl HcOptimalCostProfile { /// this entry — there is no runtime peer. /// /// The `debug_assert!(S::USE_BT, …)` enforces that - /// `MAX_CHAIN_DEPTH` / `SUFFICIENT_MATCH_LEN` are only consulted + /// `SUFFICIENT_MATCH_LEN` is only consulted /// for BT-walking strategies, since non-BT strategies /// (`Fast` / `Dfast` / `Greedy` / `Lazy`) carry placeholder - /// values for those consts — see the `MAX_CHAIN_DEPTH` doc + /// values for the BT consts; see the `MAX_CHAIN_DEPTH` doc /// comment on each of those strategy types. #[inline] pub(crate) fn const_for_strategy() -> Self { @@ -468,14 +468,14 @@ impl HcOptimalCostProfile { profile is only meaningful when the BT walker is active.", ); Self { - max_chain_depth: S::MAX_CHAIN_DEPTH, sufficient_match_len: S::SUFFICIENT_MATCH_LEN, - accurate: S::ACCURATE_PRICE, favor_small_offsets: S::FAVOR_SMALL_OFFSETS, } } - pub(crate) fn literal_price(&self, stats: &HcOptState, byte: u8) -> u32 { + /// `ACCURATE` is the strategy's price mode (upstream `optLevel >= 1`), a + /// const so the parser's per-candidate prices carry no weight-mode branch. + pub(crate) fn literal_price(&self, stats: &HcOptState, byte: u8) -> u32 { if !stats.literals_compressed() { return 8 * HC_BITCOST_MULTIPLIER; } @@ -487,28 +487,32 @@ impl HcOptimalCostProfile { // final subtract never underflows. debug_assert!(stats.lit_sum_base_price >= HC_BITCOST_MULTIPLIER); let lit_max = stats.lit_sum_base_price - HC_BITCOST_MULTIPLIER; - let mut lit_weight = HcOptState::weight(stats.lit_freq[byte as usize], self.accurate); + let mut lit_weight = HcOptState::weight(stats.lit_freq[byte as usize], ACCURATE); if lit_weight > lit_max { lit_weight = lit_max; } stats.lit_sum_base_price - lit_weight } - pub(crate) fn lit_length_price(&self, stats: &HcOptState, lit_len: usize) -> u32 { + pub(crate) fn lit_length_price( + &self, + stats: &HcOptState, + lit_len: usize, + ) -> u32 { if lit_len == HC_BLOCKSIZE_MAX { // Upstream zstd parity: ZSTD_litLengthPrice() handles the non-representable // BLOCKSIZE_MAX literal-length by charging one extra bit over the // largest encodable litLength symbol. return HC_BITCOST_MULTIPLIER - + self.lit_length_price(stats, HC_BLOCKSIZE_MAX.saturating_sub(1)); + + self.lit_length_price::(stats, HC_BLOCKSIZE_MAX.saturating_sub(1)); } if matches!(stats.price_type, HcOptPriceType::Predefined) { - return HcOptState::weight(lit_len as u32, self.accurate); + return HcOptState::weight(lit_len as u32, ACCURATE); } // ll_bits ≤ 16 ⇒ ll_bits * 256 ≤ 4096, sum no overflow. let (ll_code, ll_bits) = HcOptState::lit_code_and_bits(lit_len); ll_bits * HC_BITCOST_MULTIPLIER + stats.lit_length_sum_base_price - - HcOptState::weight(stats.lit_length_freq[ll_code], self.accurate) + - HcOptState::weight(stats.lit_length_freq[ll_code], ACCURATE) } #[inline(always)] @@ -533,18 +537,22 @@ impl HcOptimalCostProfile { } #[inline(always)] - pub(crate) fn match_length_price(&self, stats: &HcOptState, match_len: usize) -> u32 { + pub(crate) fn match_length_price( + &self, + stats: &HcOptState, + match_len: usize, + ) -> u32 { // Upstream zstd parity: mlBase = match_len - MINMATCH; callers guarantee // match_len ≥ HC_FORMAT_MINMATCH. ml_bits ≤ 16, * 256 ≤ 4096. debug_assert!(match_len >= HC_FORMAT_MINMATCH); let ml_base = match_len - HC_FORMAT_MINMATCH; if matches!(stats.price_type, HcOptPriceType::Predefined) { - return HcOptState::weight(ml_base as u32, self.accurate); + return HcOptState::weight(ml_base as u32, ACCURATE); } let (ml_code, ml_bits) = HcOptState::ml_code_and_bits(match_len); ml_bits * HC_BITCOST_MULTIPLIER + (stats.match_length_sum_base_price - - HcOptState::weight(stats.match_length_freq[ml_code], self.accurate)) + - HcOptState::weight(stats.match_length_freq[ml_code], ACCURATE)) } #[inline(always)] diff --git a/zstd/src/encoding/cparams.rs b/zstd/src/encoding/cparams.rs index b250ce942..e87f5df63 100644 --- a/zstd/src/encoding/cparams.rs +++ b/zstd/src/encoding/cparams.rs @@ -372,24 +372,6 @@ fn get_cparams_mode( adjust_cparams(cp, src_size_hint, dict_size, create_cdict) } -/// Public `ZSTD_getCParams` entry: maps `src_size_hint == 0` to UNKNOWN, -/// matching upstream exactly. The C-reference comparison surface (`zz_cparams` -/// validates it byte-for-byte against C `ZSTD_getCParams`); the encoder sizes -/// its own tables from [`default_cparams`] + [`create_cdict_table_logs`]. -#[cfg(feature = "bench-internals")] -pub(crate) fn get_cparams_public( - compression_level: i32, - src_size_hint: u64, - dict_size: usize, -) -> CParams { - let src = if src_size_hint == 0 { - CONTENTSIZE_UNKNOWN - } else { - src_size_hint - }; - get_cparams(compression_level, src, dict_size) -} - /// The `(hash_log, chain_log)` a dictionary's prepared match-finder tables get /// under `ZSTD_cpm_createCDict` — the single source for the CDict table /// geometry (mirrors `ZSTD_adjustCParams_internal` with an unknown source, so diff --git a/zstd/src/encoding/fastpath/mod.rs b/zstd/src/encoding/fastpath/mod.rs index 2d75a506e..b8bd7c915 100644 --- a/zstd/src/encoding/fastpath/mod.rs +++ b/zstd/src/encoding/fastpath/mod.rs @@ -51,8 +51,8 @@ //! //! Week 1 (this commit): module scaffold + dispatcher skeleton. //! Week 2a: match-length / common-prefix-len + `count_match_from_indices`. -//! Week 3a: BT walk (`bt_insert_step_no_rebase`, -//! `bt_insert_and_collect_matches`) + HC chain walk. +//! Week 3a: BT walk (`bt_insert_range`, the collect-matches body) + HC +//! chain walk. //! Week 3b: optimal parser DP (`build_optimal_plan_impl` + price helpers). //! Week 4: entropy encoders (FSE `encode_interleaved`, Huff0 `encode_stream`). //! Week 5-6: bench vs `perf/pre-intrinsics-refactor-baseline` tag, profile, diff --git a/zstd/src/encoding/fastpath/neon.rs b/zstd/src/encoding/fastpath/neon.rs index f93fa327b..7283ffdd4 100644 --- a/zstd/src/encoding/fastpath/neon.rs +++ b/zstd/src/encoding/fastpath/neon.rs @@ -9,13 +9,40 @@ #![cfg(all(target_arch = "aarch64", target_endian = "little"))] -use core::arch::aarch64::{uint8x16_t, vceqq_u8, vgetq_lane_u64, vld1q_u8, vreinterpretq_u64_u8}; +use core::arch::aarch64::{ + uint8x16_t, vandq_u8, vceqq_u8, vget_lane_u64, vld1q_u8, vminvq_u8, vreinterpret_u64_u8, + vreinterpretq_u16_u8, vshrn_n_u16, +}; use super::scalar; -/// 16-byte NEON vector prefix-length probe. Returns the number of leading -/// equal bytes that fit in whole 16-byte chunks; the caller (or the wrapper -/// below) handles the scalar tail. +/// Index of the first unequal byte of a 16-byte `vceqq_u8` result that has +/// one. NEON has no byte mask move: narrowing each 16-bit lane by four keeps a +/// nibble per byte, in order, so the first zero nibble is the first mismatch. +#[target_feature(enable = "neon")] +#[inline] +fn first_unequal(eq: uint8x16_t) -> usize { + let nibbles = vget_lane_u64( + vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(eq), 4)), + 0, + ); + ((!nibbles).trailing_zeros() / 4) as usize +} + +/// NEON vector prefix-length probe. Returns the number of leading equal bytes +/// that fit in whole 16-byte chunks; the caller (or the wrapper below) handles +/// the scalar tail. +/// +/// Compares 32 bytes a step with one branch: the two equality masks are +/// joined and their minimum lane tested, and the mismatch is only located on +/// the step that has one. +/// +/// Measured on M1 against the 16-byte step with two lane tests it replaced, +/// interleaved: the legacy dictionary trainer, whose comparisons run long, +/// 4.96-5.98 s -> 4.19-4.41 s wall (zstd 1.5.7: 4.35-4.67 s); the level-19 +/// encoder on z000033 1.664-1.691 -> 1.613-1.648 G cycles (libzstd: +/// 1.546-1.617 G) while a decode-only control moved 3.458-3.475 -> +/// 3.421-3.457 G. /// /// # Safety /// `lhs` / `rhs` must point to at least `max` initialized bytes. NEON must be @@ -25,20 +52,25 @@ use super::scalar; #[inline] pub(crate) unsafe fn prefix_len_simd(lhs: *const u8, rhs: *const u8, max: usize) -> usize { let mut off = 0usize; - while off + 16 <= max { - let a: uint8x16_t = unsafe { vld1q_u8(lhs.add(off)) }; - let b: uint8x16_t = unsafe { vld1q_u8(rhs.add(off)) }; - let eq = vceqq_u8(a, b); - let lanes = vreinterpretq_u64_u8(eq); - let low = vgetq_lane_u64(lanes, 0); - if low != u64::MAX { - let diff = low ^ u64::MAX; - return off + scalar::mismatch_byte_index(diff as usize); + while off + 32 <= max { + let (eq0, eq1) = unsafe { + ( + vceqq_u8(vld1q_u8(lhs.add(off)), vld1q_u8(rhs.add(off))), + vceqq_u8(vld1q_u8(lhs.add(off + 16)), vld1q_u8(rhs.add(off + 16))), + ) + }; + if vminvq_u8(vandq_u8(eq0, eq1)) != u8::MAX { + if vminvq_u8(eq0) != u8::MAX { + return off + first_unequal(eq0); + } + return off + 16 + first_unequal(eq1); } - let high = vgetq_lane_u64(lanes, 1); - if high != u64::MAX { - let diff = high ^ u64::MAX; - return off + 8 + scalar::mismatch_byte_index(diff as usize); + off += 32; + } + if off + 16 <= max { + let eq = unsafe { vceqq_u8(vld1q_u8(lhs.add(off)), vld1q_u8(rhs.add(off))) }; + if vminvq_u8(eq) != u8::MAX { + return off + first_unequal(eq); } off += 16; } diff --git a/zstd/src/encoding/fastpath/neon/tests.rs b/zstd/src/encoding/fastpath/neon/tests.rs index 30f99935e..6cb0e77be 100644 --- a/zstd/src/encoding/fastpath/neon/tests.rs +++ b/zstd/src/encoding/fastpath/neon/tests.rs @@ -14,6 +14,26 @@ fn neon_prefix_len_matches_scalar_on_long_run() { assert_eq!(neon, 25); } +/// Every length up to three 32-byte steps and every mismatch position in it, +/// or none: each half of a step, the lone 16-byte step after the loop and the +/// scalar tail all have to find the first unequal byte the scalar kernel finds. +#[test] +fn neon_prefix_len_matches_scalar_at_every_mismatch_position() { + let a: Vec = (0..100u8).map(|i| i.wrapping_mul(37)).collect(); + for max in 0..=a.len() { + for mismatch in (0..max).map(Some).chain([None]) { + let mut b = a.clone(); + if let Some(at) = mismatch { + b[at] ^= 0x5A; + } + let neon = unsafe { common_prefix_len_ptr(a.as_ptr(), b.as_ptr(), max) }; + let scl = unsafe { scalar::common_prefix_len_ptr(a.as_ptr(), b.as_ptr(), max) }; + assert_eq!(neon, scl, "max {max}, mismatch at {mismatch:?}"); + assert_eq!(neon, mismatch.unwrap_or(max)); + } + } +} + #[test] fn neon_handles_short_input() { let a = b"abc"; diff --git a/zstd/src/encoding/frame_compressor.rs b/zstd/src/encoding/frame_compressor.rs index 8bb03cbf7..f0393ca3a 100644 --- a/zstd/src/encoding/frame_compressor.rs +++ b/zstd/src/encoding/frame_compressor.rs @@ -443,7 +443,7 @@ pub(crate) type SharedFseTable = alloc::rc::Rc; /// reference counts, then whatever padding the table's alignment adds. One /// allocation holds both, so a caller sizing a context is told about the whole /// of it rather than the payload alone. -const fn shared_table_overhead() -> usize { +pub(crate) const fn shared_table_overhead() -> usize { let counts = 2 * core::mem::size_of::(); let align = core::mem::align_of::(); counts.div_ceil(align) * align diff --git a/zstd/src/encoding/frame_compressor/tests.rs b/zstd/src/encoding/frame_compressor/tests.rs index 4ccfdb2b2..f1d2227d0 100644 --- a/zstd/src/encoding/frame_compressor/tests.rs +++ b/zstd/src/encoding/frame_compressor/tests.rs @@ -1992,7 +1992,7 @@ fn frame_emit_info_decompressed_ranges_match_decoded_output() { let data = emit_info_fixture_data(); // Cover both the single-block-per-chunk path (Default) and the - // Level(16..=22) post-split path (multiple physical partitions per + // optimal-band post-split path (multiple physical partitions per // input chunk), since lsm-tree compresses at zstd:22 and post-split // is the riskiest capture site (per-partition `src_size`). for level in [ diff --git a/zstd/src/encoding/hc/generator.rs b/zstd/src/encoding/hc/generator.rs index 3d2d19844..dcd390220 100644 --- a/zstd/src/encoding/hc/generator.rs +++ b/zstd/src/encoding/hc/generator.rs @@ -78,19 +78,25 @@ pub(crate) struct HcMatchGenerator { // the top of this file bring them back into scope so the existing // methods on `HcMatchGenerator` compile unchanged. -/// `bt_insert_step_no_rebase` body parameterized over the per-CPU -/// `count_match_from_indices` symbol. Each kernel-specific wrapper invokes -/// the macro with its own `fastpath::::count_match_from_indices` -/// path so the call resolves inside the wrapper's `#[target_feature]` -/// umbrella and inlines instead of paying the function-call ABI per BT walk -/// iteration. Used only by `HcMatchGenerator` BT walk wrappers below. +/// Binary-tree insertion of the positions `[$from, $stop)`, parameterized over +/// the per-CPU `count_match_from_indices` symbol so the compare inlines under +/// each kernel wrapper's `#[target_feature]` umbrella. Evaluates to the cursor +/// after the last insertion, which a long match can carry past `$stop`: the +/// positions it covers stay out of the tree, as upstream's `ZSTD_updateTree` +/// leaves them (`idx += ZSTD_insertBt1(...)`, no clamp to the target). +/// +/// One call inserts a whole run, the shape of upstream's +/// `ZSTD_updateTree_internal` with `ZSTD_insertBt1` inlined into its loop: +/// the tables, the mask, the window floor (a function of `$target_abs`, fixed +/// for the run) and the coordinate biases are resolved once per run, not once +/// per position. The parser's catch-up inserts a couple of positions per run, +/// and resolving all of that per position cost more than the tree walk. /// /// Crate-private: the macro body references private `encoding::*` /// modules via `$crate::...`, so it is unusable downstream and is /// re-exported only inside this crate via `pub(crate) use` below. -macro_rules! bt_insert_step_no_rebase_body { - ($table:expr, $search_depth:expr, $abs_pos:ident, $current_abs_end:ident, $target_abs:ident, $cmf:path) => {{ - let idx = $abs_pos - $table.history_abs_start; +macro_rules! bt_insert_range_body { + ($table:expr, $search_depth:expr, $from:ident, $stop:ident, $current_abs_end:ident, $target_abs:ident, $cmf:path) => {{ // Borrowed-aware live region (owned: `history[history_start..]`; // borrowed: the in-place input `[0, block_end)`). Reborrow-then-raw-ptr // so the slice holds NO borrow and coexists with the `&mut $table` @@ -99,20 +105,12 @@ macro_rules! bt_insert_step_no_rebase_body { let lh = $table.live_history(); core::slice::from_raw_parts(lh.as_ptr(), lh.len()) }; - if idx + 8 > concat.len() { - return 1; - } - debug_assert!( - $abs_pos <= $current_abs_end, - "BT walker called past current block end" - ); - let tail_limit = $current_abs_end - $abs_pos; - let hash = $crate::encoding::match_table::storage::MatchTable::hash_position_at( - concat, - idx, - $table.hash_log, - $table.search_mls, - ); + let hist_start = $table.history_abs_start; + let hash_log = $table.hash_log; + let search_mls = $table.search_mls; + let position_base = $table.position_base; + let index_shift = $table.index_shift; + let search_depth = $search_depth; // Upstream holds `U32* const hashTable = ms->hashTable` for the whole // body (zstd_opt.c:449). Ours re-derived it from the shared table // buffer at every use, and each re-derivation is a bounds-checked @@ -125,210 +123,244 @@ macro_rules! bt_insert_step_no_rebase_body { // walk). Both bases come out of ONE split borrow: taking the second // through its own `&mut` reslice reborrows the whole buffer, which // invalidates a pointer already taken from the first. - debug_assert_eq!($table.hash_table().len(), 1usize << $table.hash_log); + debug_assert_eq!($table.hash_table().len(), 1usize << hash_log); debug_assert_eq!($table.chain_table().len(), 2 << $table.bt_log()); - debug_assert!(hash < 1usize << $table.hash_log); let (hash_ptr, chain_ptr) = { let (hash_table, chain_table) = $table.hash_and_chain_mut(); (hash_table.as_mut_ptr(), chain_table.as_mut_ptr()) }; - // Prefetch the hash bucket now. For the large L16+ hash table over - // high-entropy input the bucket is L3/DRAM-cold, and unlike upstream's - // monolithic ZSTD_btGetAllMatches (which overlaps this miss with its - // inline rep/hash3 prologue) the read+write of `hash_table[hash]` - // below is reached with nothing to hide it behind — it stalled a large - // share of this function's cycles. Issuing the hint here lets the miss - // overlap the address setup that follows. - #[cfg(all( - target_feature = "sse", - any(target_arch = "x86", target_arch = "x86_64") - ))] - { - #[cfg(target_arch = "x86")] - use core::arch::x86::{_MM_HINT_T0, _mm_prefetch}; - #[cfg(target_arch = "x86_64")] - use core::arch::x86_64::{_MM_HINT_T0, _mm_prefetch}; - // SAFETY: prefetch is a hint that never faults; `hash` indexes - // `hash_table` directly below, so it is in bounds. - unsafe { - _mm_prefetch(hash_ptr.add(hash).cast(), _MM_HINT_T0); - } - // Prefetch the NEXT position's bucket too. The optimal-parser DP - // advances one position per iteration, so this miss is issued a - // full BT walk plus the next iteration's pre-collect work ahead of - // the collect that will read it — far more lead than the same-call - // hint above, enough to hide the full DRAM latency. - if idx + 1 + 8 <= concat.len() { - let hash_next = - $crate::encoding::match_table::storage::MatchTable::hash_position_at( - concat, - idx + 1, - $table.hash_log, - $table.search_mls, - ); - // SAFETY: prefetch never faults; an out-of-range index is a - // harmless no-op hint. - unsafe { - _mm_prefetch(hash_ptr.add(hash_next).cast(), _MM_HINT_T0); - } - } - } - // Total, not tested: the block was armed before the parse began. - let stored = $table.relative_position_armed($abs_pos) + 1; let bt_mask = $table.bt_mask(); - // `abs_pos < bt_mask` legitimately happens for the first BT walk of - // a fresh frame (bt_low effectively "no floor"). Saturating keeps - // the floor at 0 so the `candidate_abs <= bt_low` check never - // triggers early; raw subtraction would underflow into a huge - // sentinel that ALWAYS triggers. - let bt_low = $abs_pos.saturating_sub(bt_mask); let window_low = $table.window_low_abs_for_target($target_abs); - // `abs_pos + 9` is safe in raw form: `MatchTable::add_data` caps - // total input at `usize::MAX - STREAM_ABS_HEADROOM` (where - // `STREAM_ABS_HEADROOM = HC_OPT_NUM + 16`), so every - // frame-lifetime absolute cursor passed to the BT walker stays - // below `usize::MAX - 9` regardless of stream length or - // pointer width. The guard is hoisted to the data-ingest - // boundary so this per-position site pays zero arithmetic - // overhead in the hot loop. - let mut match_end_abs = $abs_pos + 9; - let mut best_len = 8usize; - let mut compares_left = $search_depth; - let mut common_length_smaller = 0usize; - let mut common_length_larger = 0usize; - let pair_idx = $table.bt_pair_index_for_abs($abs_pos); - let mut smaller_slot = pair_idx; - let mut larger_slot = pair_idx + 1; - // SAFETY: `hash` is masked to `hash_log` bits and the table is - // `1 << hash_log` slots wide (both asserted at `hash_ptr`), so the slot - // is in range by construction. Upstream reads and writes the same slot - // through its own raw `hashTable`. - let mut match_stored = unsafe { *hash_ptr.add(hash) }; - unsafe { *hash_ptr.add(hash) = stored }; + // The walk carries one coordinate, the stored index, as the collect body + // does and as upstream carries `matchIndex`: the window bound becomes one + // unsigned range test on it (HC_EMPTY, 0, decodes below the window and + // ends the walk), and the absolute position, the history index and the + // pair slot are each one add of a bias taken here, instead of reloading + // `position_base` / `index_shift` / `history_abs_start` through the + // table on every node. See the collect body for the derivation. + let abs_bias = position_base.wrapping_sub(1).wrapping_sub(index_shift); + let win_off = abs_bias.wrapping_sub(window_low); + let idx_bias = abs_bias.wrapping_sub(hist_start); + let bt_bias = position_base.wrapping_sub(1); + let mut pos = $from; + while pos < $stop { + let forward: usize = 'insert: { + let abs_pos = pos; + let idx = abs_pos - hist_start; + if idx + 8 > concat.len() { + break 'insert 1; + } + debug_assert!( + abs_pos <= $current_abs_end, + "BT walker called past current block end" + ); + let tail_limit = $current_abs_end - abs_pos; + let hash = $crate::encoding::match_table::storage::MatchTable::hash_position_at( + concat, idx, hash_log, search_mls, + ); + debug_assert!(hash < 1usize << hash_log); + // Prefetch the hash bucket now. For the large L16+ hash table over + // high-entropy input the bucket is L3/DRAM-cold, and unlike upstream's + // monolithic ZSTD_btGetAllMatches (which overlaps this miss with its + // inline rep/hash3 prologue) the read+write of `hash_table[hash]` + // below is reached with nothing to hide it behind — it stalled a large + // share of this function's cycles. Issuing the hint here lets the miss + // overlap the address setup that follows. + #[cfg(all( + target_feature = "sse", + any(target_arch = "x86", target_arch = "x86_64") + ))] + { + #[cfg(target_arch = "x86")] + use core::arch::x86::{_MM_HINT_T0, _mm_prefetch}; + #[cfg(target_arch = "x86_64")] + use core::arch::x86_64::{_MM_HINT_T0, _mm_prefetch}; + // SAFETY: prefetch is a hint that never faults; `hash` indexes + // `hash_table` directly below, so it is in bounds. + unsafe { + _mm_prefetch(hash_ptr.add(hash).cast(), _MM_HINT_T0); + } + // Prefetch the NEXT position's bucket too. The optimal-parser DP + // advances one position per iteration, so this miss is issued a + // full BT walk plus the next iteration's pre-collect work ahead of + // the collect that will read it — far more lead than the same-call + // hint above, enough to hide the full DRAM latency. + if idx + 1 + 8 <= concat.len() { + let hash_next = + $crate::encoding::match_table::storage::MatchTable::hash_position_at( + concat, + idx + 1, + hash_log, + search_mls, + ); + // SAFETY: prefetch never faults; an out-of-range index is a + // harmless no-op hint. + unsafe { + _mm_prefetch(hash_ptr.add(hash_next).cast(), _MM_HINT_T0); + } + } + } + // Total, not tested: the block was armed before the parse began + // (`MatchTable::relative_position_armed`, with its fields hoisted). + debug_assert!($table.can_skip_rebase_check(abs_pos)); + let stored = ((abs_pos - position_base + index_shift) as u32) + 1; + // `abs_pos < bt_mask` legitimately happens for the first BT walk of + // a fresh frame (bt_low effectively "no floor"). Saturating keeps + // the floor at 0 so the `candidate_abs <= bt_low` check never + // triggers early; raw subtraction would underflow into a huge + // sentinel that ALWAYS triggers. + let bt_low = abs_pos.saturating_sub(bt_mask); + // The window floor follows the tree update's target, which may lie past + // this position by more than the window; nothing is in range then, as + // upstream's `matchIndex >= windowLow` finds on its first test. + let win_range = if window_low < abs_pos { + abs_pos - window_low + } else { + 0 + }; + // `abs_pos + 9` is safe in raw form: `MatchTable::add_data` caps + // total input at `usize::MAX - STREAM_ABS_HEADROOM` (where + // `STREAM_ABS_HEADROOM = HC_OPT_NUM + 16`), so every + // frame-lifetime absolute cursor passed to the BT walker stays + // below `usize::MAX - 9` regardless of stream length or + // pointer width. The guard is hoisted to the data-ingest + // boundary so this per-position site pays zero arithmetic + // overhead in the hot loop. + let mut match_end_abs = abs_pos + 9; + let mut best_len = 8usize; + let mut compares_left = search_depth; + let mut common_length_smaller = 0usize; + let mut common_length_larger = 0usize; + // `MatchTable::bt_pair_index_for_abs` with the shift and mask hoisted. + let pair_idx = 2 * (abs_pos.wrapping_add(index_shift) & bt_mask); + let mut smaller_slot = pair_idx; + let mut larger_slot = pair_idx + 1; + // SAFETY: `hash` is masked to `hash_log` bits and the table is + // `1 << hash_log` slots wide (both asserted at `hash_ptr`), so the slot + // is in range by construction. Upstream reads and writes the same slot + // through its own raw `hashTable`. + let mut match_stored = unsafe { *hash_ptr.add(hash) }; + unsafe { *hash_ptr.add(hash) = stored }; - while compares_left > 0 { - if match_stored == $crate::encoding::match_table::storage::HC_EMPTY { - break; - } - // Reject stale post-rebase slots whose pre-shift position is below - // `index_shift` explicitly. A `wrapping_sub` maps such a slot to a - // near-`usize::MAX` value that the `>= abs_pos` test only rejects - // while `abs_pos` is far from the integer ceiling; on a - // long-running rebased stream (reachable on 32-bit) `abs_pos` can - // approach the ceiling and the wrapped value can land back inside - // `[window_low, abs_pos)`. Ending the walk on the underflow avoids - // that. `match_stored != HC_EMPTY` here, so the `- 1` cannot - // underflow. The shift is taken off the stored index before the - // floor is added, because on a 32-bit word the floor plus a stored - // index need not fit; a slot under the shift decodes below - // `position_base`, which the window floor rejects anyway. - let match_relative = match_stored as usize - 1; - if match_relative < $table.index_shift { - break; - } - let candidate_abs = $table.position_base + (match_relative - $table.index_shift); - if candidate_abs < window_low || candidate_abs >= $abs_pos { - break; - } - compares_left -= 1; + while compares_left > 0 && (match_stored as usize).wrapping_add(win_off) < win_range + { + compares_left -= 1; + let stored = match_stored as usize; + // A slot written under an earlier encoding would sit below the + // shift; the block was armed, and a rebase rewrites every slot, so + // none reaches an in-window test. + debug_assert!(stored > index_shift); + let candidate_abs = stored.wrapping_add(abs_bias); + debug_assert!(candidate_abs >= window_low && candidate_abs < abs_pos); + // `2*((candidate_abs + index_shift) & bt_mask)`, with the shift + // folded: `candidate_abs + index_shift == stored + bt_bias`. + let next_pair_idx = 2 * (stored.wrapping_add(bt_bias) & bt_mask); + // SAFETY: `next_pair_idx (+1)` = `2*(candidate_abs & bt_mask) (+1)` + // ≤ `chain_table.len()-1`; `chain_ptr` is the hoisted live base, + // table not realloc'd during the walk. + let next_smaller = unsafe { *chain_ptr.add(next_pair_idx) }; + let next_larger = unsafe { *chain_ptr.add(next_pair_idx + 1) }; + let seed_len = common_length_smaller.min(common_length_larger); + let candidate_idx = stored.wrapping_add(idx_bias); + // SAFETY: BT walk invariant — `candidate_idx + tail_limit ≤ + // concat.len()` since the candidate is within + // `[history_abs_start, abs_pos)` and `tail_limit ≤ + // current_abs_end - abs_pos`. + let match_len = + unsafe { $cmf(concat, idx, candidate_idx, tail_limit, seed_len) }; - let next_pair_idx = $table.bt_pair_index_for_abs(candidate_abs); - // SAFETY: `next_pair_idx (+1)` = `2*(candidate_abs & bt_mask) (+1)` - // ≤ `chain_table.len()-1`; `chain_ptr` is the hoisted live base, - // table not realloc'd during the walk. - let next_smaller = unsafe { *chain_ptr.add(next_pair_idx) }; - let next_larger = unsafe { *chain_ptr.add(next_pair_idx + 1) }; - let seed_len = common_length_smaller.min(common_length_larger); - let candidate_idx = candidate_abs - $table.history_abs_start; - // SAFETY: BT walk invariant — `candidate_idx + tail_limit ≤ - // concat.len()` since the candidate is within - // `[history_abs_start, abs_pos)` and `tail_limit ≤ - // current_abs_end - abs_pos`. - let match_len = unsafe { $cmf(concat, idx, candidate_idx, tail_limit, seed_len) }; + if match_len > best_len { + best_len = match_len; + // `candidate_abs + match_len <= current_abs_end` by BT walk + // invariant — `match_len <= tail_limit = current_abs_end - + // abs_pos` and `candidate_abs < abs_pos`. + let candidate_end = candidate_abs + match_len; + if candidate_end > match_end_abs { + match_end_abs = candidate_end; + } + } - if match_len > best_len { - best_len = match_len; - // `candidate_abs + match_len <= current_abs_end` by BT walk - // invariant — `match_len <= tail_limit = current_abs_end - - // abs_pos` and `candidate_abs < abs_pos`. - let candidate_end = candidate_abs + match_len; - if candidate_end > match_end_abs { - match_end_abs = candidate_end; - } - } + if match_len >= tail_limit { + break; + } - if match_len >= tail_limit { - break; - } + let candidate_next = candidate_idx + match_len; + let current_next = idx + match_len; + // SAFETY: first-differing positions after a match_len-long prefix; + // match_len < tail_limit (break above) + BT-walk bound + // idx/candidate_idx + tail_limit <= concat.len() keep both in range. + if unsafe { + *concat.get_unchecked(candidate_next) < *concat.get_unchecked(current_next) + } { + // SAFETY: `smaller_slot` holds a valid pair index (init + // `pair_idx`, updated to `next_pair_idx + 1`); the `usize::MAX` + // sentinel is set only just before `break`, never written here. + unsafe { *chain_ptr.add(smaller_slot) = match_stored }; + common_length_smaller = match_len; + if candidate_abs <= bt_low { + smaller_slot = usize::MAX; + break; + } + smaller_slot = next_pair_idx + 1; + match_stored = next_larger; + } else { + // SAFETY: as above for `larger_slot`. + unsafe { *chain_ptr.add(larger_slot) = match_stored }; + common_length_larger = match_len; + if candidate_abs <= bt_low { + larger_slot = usize::MAX; + break; + } + larger_slot = next_pair_idx; + match_stored = next_smaller; + } + } - let candidate_next = candidate_idx + match_len; - let current_next = idx + match_len; - // SAFETY: first-differing positions after a match_len-long prefix; - // match_len < tail_limit (break above) + BT-walk bound - // idx/candidate_idx + tail_limit <= concat.len() keep both in range. - if unsafe { - *concat.get_unchecked(candidate_next) < *concat.get_unchecked(current_next) - } { - // SAFETY: `smaller_slot` holds a valid pair index (init - // `pair_idx`, updated to `next_pair_idx + 1`); the `usize::MAX` - // sentinel is set only just before `break`, never written here. - unsafe { *chain_ptr.add(smaller_slot) = match_stored }; - common_length_smaller = match_len; - if candidate_abs <= bt_low { - smaller_slot = usize::MAX; - break; + // SAFETY: both slots, when not the `usize::MAX` sentinel, hold valid + // pair indices into the hoisted `chain_table` base. + if smaller_slot != usize::MAX { + unsafe { + *chain_ptr.add(smaller_slot) = + $crate::encoding::match_table::storage::HC_EMPTY + }; } - smaller_slot = next_pair_idx + 1; - match_stored = next_larger; - } else { - // SAFETY: as above for `larger_slot`. - unsafe { *chain_ptr.add(larger_slot) = match_stored }; - common_length_larger = match_len; - if candidate_abs <= bt_low { - larger_slot = usize::MAX; - break; + if larger_slot != usize::MAX { + unsafe { + *chain_ptr.add(larger_slot) = + $crate::encoding::match_table::storage::HC_EMPTY + }; } - larger_slot = next_pair_idx; - match_stored = next_smaller; - } - } - // SAFETY: both slots, when not the `usize::MAX` sentinel, hold valid - // pair indices into the hoisted `chain_table` base. - if smaller_slot != usize::MAX { - unsafe { - *chain_ptr.add(smaller_slot) = $crate::encoding::match_table::storage::HC_EMPTY - }; - } - if larger_slot != usize::MAX { - unsafe { - *chain_ptr.add(larger_slot) = $crate::encoding::match_table::storage::HC_EMPTY + let speed_positions = if best_len > 384 { + (best_len - 384).min(192) + } else { + 0 + }; + // `match_end_abs` is initialized to `abs_pos + 9` and is only + // reassigned inside the `candidate_end > match_end_abs` branch + // above. So even though an individual `candidate_end = + // candidate_abs + match_len` can land below `abs_pos` (the + // candidate sits earlier in history and the match runs short), + // the variable itself never drops below its initial value. + // That gives `match_end_abs ≥ abs_pos + 9 > abs_pos + 8` as a + // loop-wide invariant, so the raw subtraction below cannot + // underflow. + speed_positions.max(match_end_abs - (abs_pos + 8)) }; + // Both terms of `forward` end inside the block (`match_end_abs <= + // current_abs_end`, and a speed skip needs a match over 384 bytes), so + // the cursor cannot overflow. + pos += forward.max(1); } - - let speed_positions = if best_len > 384 { - (best_len - 384).min(192) - } else { - 0 - }; - // `match_end_abs` is initialized to `abs_pos + 9` and is only - // reassigned inside the `candidate_end > match_end_abs` branch - // above. So even though an individual `candidate_end = - // candidate_abs + match_len` can land below `abs_pos` (the - // candidate sits earlier in history and the match runs short), - // the variable itself never drops below its initial value. - // That gives `match_end_abs ≥ abs_pos + 9 > abs_pos + 8` as a - // loop-wide invariant, so the raw subtraction below cannot - // underflow. - speed_positions.max(match_end_abs - ($abs_pos + 8)) + pos }}; } -pub(crate) use bt_insert_step_no_rebase_body; +pub(crate) use bt_insert_range_body; /// `hash3_candidate` body parameterized over the per-CPU /// `common_prefix_len_ptr` symbol. The hash3 probe checks one candidate per /// position when invoked, so the per-call ABI savings compound across the -/// segment. Crate-private (see `bt_insert_step_no_rebase_body!`). +/// segment. Crate-private (see `bt_insert_range_body!`). macro_rules! hash3_candidate_body { ( $table:expr, @@ -392,7 +424,7 @@ pub(crate) use hash3_candidate_body; /// /// The callback `f` runs in the wrapper's umbrella context too, so closures /// that capture mutable state still work (FnMut). Crate-private -/// (see `bt_insert_step_no_rebase_body!`). +/// (see `bt_insert_range_body!`). macro_rules! for_each_repcode_candidate_body { ( $table:expr, @@ -480,13 +512,8 @@ macro_rules! for_each_repcode_candidate_body { } pub(crate) use for_each_repcode_candidate_body; -/// `bt_insert_and_collect_matches` body parameterized over the per-CPU -/// `count_match_from_indices` symbol. Same shape as -/// [`bt_insert_step_no_rebase_body`] — picks up the matching kernel through -/// `$cmf` so the per-iteration vector probe inlines under the wrapper's -/// `target_feature` umbrella. Returns nothing (matches the original method). -/// Crate-private (see `bt_insert_step_no_rebase_body!`). -/// One repeat-offset probe, expanded per slot. +/// One repeat-offset probe, expanded per slot. Crate-private (see +/// `bt_insert_range_body!`). /// /// The repeat loop runs three or four times with the slot known at each /// expansion, so it is unrolled rather than counted: the counter, its bound and @@ -581,7 +608,7 @@ macro_rules! bt_insert_and_collect_matches_body { $best_len_for_skip:ident, $out:ident, $reps:ident, - $lit_len:ident, + $ll0:ident, $use_hash3:expr, $cpl:path, $cmf:path $(,)? @@ -686,7 +713,7 @@ macro_rules! bt_insert_and_collect_matches_body { ) }; } - if $lit_len == 0 { + if $ll0 { probe!($reps[1] as usize); probe!($reps[2] as usize); probe!(($reps[0] as usize).wrapping_sub(1)); @@ -876,10 +903,16 @@ macro_rules! bt_insert_and_collect_matches_body { } } } + // The tree's coordinates are fixed for the block and were taken before + // its parse, so none is derived again per position. + let coords = $table.block_coords; + debug_assert_eq!(coords, $table.bt_coords(), "block coordinates are stale"); // Total, not tested: the block was armed before the parse began. - let stored = $table.relative_position_armed($abs_pos) + 1; - let bt_mask = $table.bt_mask(); - // See `bt_insert_step_no_rebase_body!`: saturating is needed for the + // `abs_pos - abs_bias` is `relative_position_armed(abs_pos) + 1`. + debug_assert!($table.can_skip_rebase_check($abs_pos)); + let stored = $abs_pos.wrapping_sub(coords.abs_bias) as u32; + let bt_mask = coords.bt_mask; + // See `bt_insert_range_body!`: saturating is needed for the // first BT walk of a fresh frame where `abs_pos < bt_mask`. let bt_low = $abs_pos.saturating_sub(bt_mask); let window_low = $table.window_low_abs_for_target($abs_pos); @@ -893,11 +926,7 @@ macro_rules! bt_insert_and_collect_matches_body { // abs_pos - window_low ⟺ s.wrapping_add(win_off) < win_range. // HC_EMPTY (s = 0) maps to base = (lowest representable abs) - 1 < // window_low, so it falls out of range and ends the walk. - let win_off = $table - .position_base - .wrapping_sub(1) - .wrapping_sub($table.index_shift) - .wrapping_sub(window_low); + let win_off = coords.abs_bias.wrapping_sub(window_low); let win_range = $abs_pos - window_low; // Decode biases: fold the per-node coordinate conversions into // loop-invariant additions. The gate-validated chain entry @@ -911,13 +940,10 @@ macro_rules! bt_insert_and_collect_matches_body { // single-coordinate equivalent. Wrapping throughout: the window gate // already proved `match_stored ∈ [window_low, abs_pos)` before decode, // mirroring the `win_off` form above. - let abs_bias = $table - .position_base - .wrapping_sub(1) - .wrapping_sub($table.index_shift); - let idx_bias = abs_bias.wrapping_sub($table.history_abs_start); - let bt_bias = $table.position_base.wrapping_sub(1); - // Raw `+ 9` is safe here — see `bt_insert_step_no_rebase_body!` + let abs_bias = coords.abs_bias; + let idx_bias = coords.idx_bias; + let bt_bias = coords.bt_bias; + // Raw `+ 9` is safe here — see `bt_insert_range_body!` // for the full discussion of the upstream `STREAM_ABS_HEADROOM` // cap in `MatchTable::add_data`. let mut match_end_abs = $abs_pos + 9; @@ -932,7 +958,10 @@ macro_rules! bt_insert_and_collect_matches_body { let mut compares_left = ($max_chain_depth).min($search_depth); let mut common_length_smaller = 0usize; let mut common_length_larger = 0usize; - let pair_idx = $table.bt_pair_index_for_abs($abs_pos); + // `bt_pair_index_for_abs(abs_pos)`: `stored + bt_bias` is + // `abs_pos + index_shift`. + let pair_idx = 2 * ((stored as usize).wrapping_add(bt_bias) & bt_mask); + debug_assert_eq!(pair_idx, $table.bt_pair_index_for_abs($abs_pos)); let mut smaller_slot = pair_idx; let mut larger_slot = pair_idx + 1; // SAFETY: `hash` is masked to `hash_log` bits and the table is diff --git a/zstd/src/encoding/hc/optimal.rs b/zstd/src/encoding/hc/optimal.rs index 95d83e77f..289db7918 100644 --- a/zstd/src/encoding/hc/optimal.rs +++ b/zstd/src/encoding/hc/optimal.rs @@ -24,6 +24,8 @@ use alloc::vec::Vec; // does not count macro-body references, so it reports every one of these as // "unused" even though each macro expansion requires them (gating or removing // any one breaks the lib build). Suppress the false positive on the group. +#[cfg(test)] +use crate::encoding::opt::types::HcCandidateQuery; #[allow(unused_imports)] use crate::encoding::{ bt::BtMatcher, @@ -37,8 +39,8 @@ use crate::encoding::{ match_table::storage::HC3_HASH_LOG, opt::ldm::{HcOptLdmState, HcRawSeqStore}, opt::types::{ - HcCandidateQuery, HcOptimalNode, HcOptimalPlanBuffers, HcOptimalPlanState, - HcOptimalSequence, MatchCandidate, + HcBlockPass, HcOptimalNode, HcOptimalPlanBuffers, HcOptimalPlanState, HcOptimalSequence, + MatchCandidate, }, }; @@ -54,7 +56,17 @@ macro_rules! build_optimal_plan_impl_body { $out:ident, $buffers:expr, $collect:ident, - $priceset:path $(,)? + $priceset:path, + // The whole block and its statistics: the traceback records each + // sequence it settles into them (upstream `ZSTD_updateStats`). + $block:ident, + $opt_state:ident, + // False for the btultra2 seed pass, which keeps only the statistics. + $keep_plan:expr, + // The block the caller wraps this body in: every exit is a `break` out + // of it with the segment's result, so the body runs inside the caller's + // segment loop rather than as a function of its own. + $seg:lifetime $(,)? ) => {{ let current_abs_end = $current_abs_start + $current_len; let min_match_len = HC_OPT_MIN_MATCH_LEN; @@ -107,6 +119,7 @@ macro_rules! build_optimal_plan_impl_body { store, price_arena, candidates_searched_at: searched_at, + pass, } = &mut *$buffers; // The node arenas are indexed through base pointers resolved once, the // way upstream zstd indexes `opt[]`: nothing in this body resizes them, @@ -199,11 +212,8 @@ macro_rules! build_optimal_plan_impl_body { $current_abs_start + skipped_literals, current_abs_end, profile.sufficient_match_len, - HcCandidateQuery { - reps: initial_reps, - lit_len: initial_litlen + skipped_literals, - ldm_candidate: None, - }, + &initial_reps, + initial_litlen + skipped_literals == 0, &mut *candidates, ) }; @@ -225,7 +235,7 @@ macro_rules! build_optimal_plan_impl_body { // Literals only: no sequence was emitted and the repcodes are // untouched, exactly as the per-literal returns left them. The // price is discarded by the caller. - return ( + break $seg ( 0u32, initial_reps, initial_litlen + skipped_literals, @@ -307,14 +317,16 @@ macro_rules! build_optimal_plan_impl_body { $current_abs_start, current_abs_end, profile.sufficient_match_len, - HcCandidateQuery { - reps: initial_reps, - lit_len: initial_litlen, - ldm_candidate: seed_ldm, - }, + &initial_reps, + initial_litlen == 0, &mut *candidates, ) }; + // The long-distance candidate joins after the search, as + // upstream's `ZSTD_optLdm_processMatchCandidate` does. + if let Some(ldm) = seed_ldm { + BtMatcher::push_ldm_candidate(&mut *candidates, ldm, min_match_len); + } } if !candidates.is_empty() { // Deferred price-cache setup: the arena slices are two disjoint @@ -347,7 +359,7 @@ macro_rules! build_optimal_plan_impl_body { // Deferred base/seed prices: only reached on a matched seed (see // the declarations above). Assign before the forward DP / seed // paths below read them. - let node0_price = BtMatcher::cached_lit_length_price( + let node0_price = BtMatcher::cached_lit_length_price::( profile, $stats, initial_litlen, @@ -363,14 +375,14 @@ macro_rules! build_optimal_plan_impl_body { ..HcOptimalNode::default() }); } - ll0_price = BtMatcher::cached_lit_length_price( + ll0_price = BtMatcher::cached_lit_length_price::( profile, $stats, 0, &mut ll_cache, ll_price_stamp, ); - ll1_price = BtMatcher::cached_lit_length_price( + ll1_price = BtMatcher::cached_lit_length_price::( profile, $stats, 1, @@ -399,7 +411,7 @@ macro_rules! build_optimal_plan_impl_body { ); let off_price = profile .offset_price_for::($stats, off_base); - let ml_price = BtMatcher::cached_match_length_price( + let ml_price = BtMatcher::cached_match_length_price::( profile, $stats, longest_len, @@ -460,7 +472,7 @@ macro_rules! build_optimal_plan_impl_body { // SAFETY: cell 0 was written by the seed above. let nodes0_price = unsafe { *node_prices }; for match_len in (start_len..=max_match_len).rev() { - let ml_price = BtMatcher::cached_match_length_price( + let ml_price = BtMatcher::cached_match_length_price::( profile, $stats, match_len, @@ -477,12 +489,11 @@ macro_rules! build_optimal_plan_impl_body { let node_price = unsafe { *node_prices.add(match_len) }; if match_len > last_pos || next_cost < node_price { unsafe { - *nodes.add(match_len) = HcOptimalNode { - off: candidate.offset as u32, - mlen: match_len as u32, - litlen: 0, - reps: initial_reps, - }; + HcOptimalNode::write_match_end( + nodes.add(match_len), + candidate.offset as u32, + match_len as u32, + ); *node_prices.add(match_len) = next_cost; } if match_len > last_pos { @@ -513,7 +524,7 @@ macro_rules! build_optimal_plan_impl_body { let lit_len = prev_node.litlen as usize + 1; let lit_price = { let bt = $self.backend.bt_mut(); - BtMatcher::cached_literal_price( + BtMatcher::cached_literal_price::( profile, $stats, $current[pos - 1], @@ -522,7 +533,7 @@ macro_rules! build_optimal_plan_impl_body { lit_price_stamp, ) }; - let ll_delta = BtMatcher::cached_lit_length_delta_price( + let ll_delta = BtMatcher::cached_lit_length_delta_price::( profile, $stats, lit_len, @@ -534,14 +545,20 @@ macro_rules! build_optimal_plan_impl_body { // below) — also the price of `prev_match`, the pre-overwrite copy. let node_pos_price = unsafe { *node_prices.add(pos) }; if lit_cost <= node_pos_price { - // An unreached cell has no node to read; the default node - // (`litlen != 0`) fails the end-of-match test below exactly - // as a reset node would. - let prev_match = if node_pos_price != u32::MAX { - unsafe { *nodes.add(pos) } - } else { - HcOptimalNode::default() - }; + // The cell being replaced, read field by field: a match + // written it without a repeat history (the pass derives that + // on arrival), so the node as a whole may not be initialised. + // An unreached cell has no node to read; a zero length fails + // the end-of-match test below exactly as a reset node would. + let (prev_match_off, prev_match_mlen, prev_match_litlen) = + if node_pos_price != u32::MAX { + unsafe { + let cell = nodes.add(pos); + ((*cell).off, (*cell).mlen, (*cell).litlen) + } + } else { + (0, 0, u32::MAX) + }; unsafe { *nodes.add(pos) = HcOptimalNode { litlen: lit_len as u32, @@ -551,14 +568,14 @@ macro_rules! build_optimal_plan_impl_body { } #[allow(clippy::collapsible_if)] if opt_level - && prev_match.mlen > 0 - && prev_match.litlen == 0 + && prev_match_mlen > 0 + && prev_match_litlen == 0 && pos < $current_len { if ll1_price < ll0_price { let next_lit_price = { let bt = $self.backend.bt_mut(); - BtMatcher::cached_literal_price( + BtMatcher::cached_literal_price::( profile, $stats, $current[pos], @@ -572,7 +589,8 @@ macro_rules! build_optimal_plan_impl_body { next_lit_price, ll1_price as i32 - ll0_price as i32, ); - let ll_delta_next = BtMatcher::cached_lit_length_delta_price( + let ll_delta_next = + BtMatcher::cached_lit_length_delta_price::( profile, $stats, lit_len + 1, @@ -585,13 +603,13 @@ macro_rules! build_optimal_plan_impl_body { let next_price = unsafe { *node_prices.add(next) }; if with1literal < with_more_literals && with1literal < next_price { // Upstream zstd parity (zstd_opt.c:1232): `cur >= prevMatch.mlen`. - debug_assert!(pos >= prev_match.mlen as usize); - let prev_pos = pos - prev_match.mlen as usize; + debug_assert!(pos >= prev_match_mlen as usize); + let prev_pos = pos - prev_match_mlen as usize; { debug_assert!(unsafe { *node_prices.add(prev_pos) } != u32::MAX); let prev_state = unsafe { *nodes.add(prev_pos) }; let (_, reps_after_match) = BtMatcher::encode_offset_with_reps( - prev_match.off, + prev_match_off, prev_state.litlen as usize, prev_state.reps, ); @@ -600,9 +618,10 @@ macro_rules! build_optimal_plan_impl_body { // joins the frontier below. unsafe { *nodes.add(next) = HcOptimalNode { - reps: reps_after_match, + off: prev_match_off, + mlen: prev_match_mlen, litlen: 1, - ..prev_match + reps: reps_after_match, }; *node_prices.add(next) = with1literal; } @@ -636,15 +655,20 @@ macro_rules! build_optimal_plan_impl_body { continue; } { - let base_node = unsafe { *nodes.add(pos) }; - if base_node.mlen > 0 && base_node.litlen == 0 { + // Field by field: a match-written cell has no repeat history + // yet, which this block is about to derive. + let (base_off, base_mlen, base_litlen) = unsafe { + let cell = nodes.add(pos); + ((*cell).off, (*cell).mlen, (*cell).litlen) + }; + if base_mlen > 0 && base_litlen == 0 { // Upstream zstd parity (zstd_opt.c:1255): `cur >= opt[cur].mlen`. - debug_assert!(pos >= base_node.mlen as usize); - let prev_pos = pos - base_node.mlen as usize; + debug_assert!(pos >= base_mlen as usize); + let prev_pos = pos - base_mlen as usize; debug_assert!(unsafe { *node_prices.add(prev_pos) } != u32::MAX); let prev_state = unsafe { *nodes.add(prev_pos) }; let (_, reps_after_match) = BtMatcher::encode_offset_with_reps( - base_node.off, + base_off, prev_state.litlen as usize, prev_state.reps, ); @@ -675,34 +699,35 @@ macro_rules! build_optimal_plan_impl_body { } let abs_pos = $current_abs_start + pos; - let ldm_candidate = if HAS_LDM { - $self.backend.bt_mut().ldm_process_match_candidate( - &mut opt_ldm, - pos, - $current_len - pos, - min_match_len, - ) - } else { - None - }; candidates.clear(); - // SAFETY: same umbrella as `$collect`. Query fields are read - // fresh here (consumed into the call's argument) so they do not - // stay live across the call; the post-call reads below are a - // separate, fresh load of the same stable `nodes[pos]`. + // SAFETY: same umbrella as `$collect`. The search reads the repeat + // history in place through `nodes[pos]`, as upstream passes + // `opt[cur].rep`, so nothing is copied into the call and nothing + // stays live across it; the post-call reads below are a fresh load + // of the same stable `nodes[pos]`. unsafe { $self.$collect::<$strategy_ty>( abs_pos, current_abs_end, profile.sufficient_match_len, - HcCandidateQuery { - reps: (*nodes.add(pos)).reps, - lit_len: (*nodes.add(pos)).litlen as usize, - ldm_candidate, - }, + &(*nodes.add(pos)).reps, + (*nodes.add(pos)).litlen == 0, &mut *candidates, ) }; + if HAS_LDM { + // Upstream `ZSTD_optLdm_processMatchCandidate`: the producer + // advances past this position and its candidate joins after + // the search. + if let Some(ldm) = $self.backend.bt_mut().ldm_process_match_candidate( + &mut opt_ldm, + pos, + $current_len - pos, + min_match_len, + ) { + BtMatcher::push_ldm_candidate(&mut *candidates, ldm, min_match_len); + } + } // Post-call reads of opt[cur]: fresh, born after `$collect`, so // never part of the cross-call live set (see memory-resident note // above). `nodes[pos]` is untouched by `$collect`. @@ -722,7 +747,7 @@ macro_rules! build_optimal_plan_impl_body { ); let off_price = profile .offset_price_for::($stats, off_base); - let ml_price = BtMatcher::cached_match_length_price( + let ml_price = BtMatcher::cached_match_length_price::( profile, $stats, longest_len, @@ -786,7 +811,7 @@ macro_rules! build_optimal_plan_impl_body { // skipped. Order-dependent, stays scalar. for match_len in (start_len..=max_match_len).rev() { let next = pos + match_len; - let ml_price = BtMatcher::cached_match_length_price( + let ml_price = BtMatcher::cached_match_length_price::( profile, $stats, match_len, @@ -801,12 +826,11 @@ macro_rules! build_optimal_plan_impl_body { let node_next_price = unsafe { *node_prices.add(next) }; if next > last_pos || next_cost < node_next_price { unsafe { - *nodes.add(next) = HcOptimalNode { - off: candidate.offset as u32, - mlen: match_len as u32, - litlen: 0, - reps: base_reps, - }; + HcOptimalNode::write_match_end( + nodes.add(next), + candidate.offset as u32, + match_len as u32, + ); *node_prices.add(next) = next_cost; } if next > last_pos { @@ -855,7 +879,6 @@ macro_rules! build_optimal_plan_impl_body { off_price, base_cost, candidate.offset as u32, - base_reps, last_pos, ) }); @@ -873,7 +896,7 @@ macro_rules! build_optimal_plan_impl_body { if last_pos == 0 { if $current_len == 0 { let price = 0u32; // deferred: node_prices[0] unset on no-match; caller discards price - return (price, initial_reps, initial_litlen, 0); + break $seg (price, initial_reps, initial_litlen, 0); } // No match at this position: it is a single literal (upstream zstd // `ZSTD_compressBlock_opt_generic` `if (!nbMatches) { ip++; }`). The @@ -890,7 +913,7 @@ macro_rules! build_optimal_plan_impl_body { // node_prices[0] is unset on a no-match seed (deferred); the caller // discards this price anyway. let price = 0u32; - return (price, initial_reps, next_litlen, 1); + break $seg (price, initial_reps, next_litlen, 1); } let target_pos = forced_end.unwrap_or(last_pos.min(frontier_limit)); @@ -906,11 +929,11 @@ macro_rules! build_optimal_plan_impl_body { unsafe { (*nodes.add(target_pos), *node_prices.add(target_pos)) } }; if last_stretch_price == u32::MAX { - return (u32::MAX, initial_reps, initial_litlen, $current_len); + break $seg (u32::MAX, initial_reps, initial_litlen, $current_len); } if last_stretch.mlen == 0 { - return ( + break $seg ( last_stretch_price, last_stretch.reps, last_stretch.litlen as usize, @@ -932,7 +955,7 @@ macro_rules! build_optimal_plan_impl_body { } else { let tail_literals = last_stretch.litlen as usize; if cur < tail_literals { - return ( + break $seg ( last_stretch_price, last_stretch.reps, tail_literals, @@ -995,6 +1018,12 @@ macro_rules! build_optimal_plan_impl_body { let mut tail_literals = initial_litlen; let mut store_pos = store_start; + // Each settled sequence goes into the statistics here, as upstream's + // traceback calls `ZSTD_updateStats` per stored sequence, instead of in + // a second pass over the plan after the segment. The repeat history + // runs from the segment's own. + let mut stats_reps = initial_reps; + let mut recorded = false; while store_pos <= store_end { let stretch = store[store_pos]; let llen = stretch.litlen as usize; @@ -1004,14 +1033,29 @@ macro_rules! build_optimal_plan_impl_body { store_pos += 1; continue; } - $out.push(HcOptimalSequence { + let sequence = HcOptimalSequence { offset: stretch.off, match_len: mlen as u32, lit_len: llen as u32, - }); + }; + if $keep_plan { + $out.push(sequence); + } + BtMatcher::record_sequence_stats( + $block, + sequence, + &mut pass.literals_cursor, + &mut stats_reps, + &mut *$opt_state, + ); + recorded = true; tail_literals = 0; store_pos += 1; } + if recorded { + $opt_state + .set_base_prices(<$strategy_ty as crate::encoding::strategy::Strategy>::ACCURATE_PRICE); + } let result = ( last_stretch_price, end_reps, @@ -1026,6 +1070,85 @@ macro_rules! build_optimal_plan_impl_body { }}; } +/// The optimal parser's pass over one block, upstream zstd +/// `ZSTD_compressBlock_opt_generic`: the segment loop around each segment's +/// forward pass and traceback ([`build_optimal_plan_impl_body!`]), whose +/// traceback also records the segment's sequences into the statistics. The +/// frame, the buffer set-up and the arguments are paid once per block rather +/// than once per segment, which on input with few matches is once per literal +/// run. `$keep_plan` is false for the btultra2 seed pass, which keeps only the +/// statistics. +macro_rules! optimal_block_body { + ( + $self:expr, + $strategy_ty:ty, + $current:ident, + $current_abs_start:ident, + $cursor:ident, + $litlen:ident, + $reps:ident, + $profile:ident, + $opt_state:ident, + $plan:ident, + $buffers:ident, + $keep_plan:expr, + $collect:ident, + $priceset:path $(,)? + ) => {{ + // The block is armed, so the tree's coordinates hold for the whole pass. + $self.table.capture_block_coords(); + // Everything the pass carries between segments lives in `$buffers.pass`, + // read at the top of a segment and written at its end, so none of it is + // live across the segment body (see `HcOptimalPlanBuffers::pass`). + $buffers.pass = HcBlockPass { + cursor: $cursor, + litlen: $litlen, + reps: $reps, + literals_cursor: 0, + }; + // The last 8 bytes are left as literals. Written as an addition on the + // cursor, which never passes the block's end, so a block shorter than + // the tail needs no floor: it simply has no segment. + let block_len = $current.len(); + while $buffers.pass.cursor + 8 < block_len { + let cursor = $buffers.pass.cursor; + let segment = &$current[cursor..]; + let segment_abs_start = $current_abs_start + cursor; + let segment_len = $current.len() - cursor; + let segment_state = HcOptimalPlanState { + block_offset: cursor, + reps: $buffers.pass.reps, + litlen: $buffers.pass.litlen, + profile: $profile, + }; + let segment_stats: &HcOptState = &*$opt_state; + let (_, end_reps, end_litlen, consumed_len) = 'segment: { + build_optimal_plan_impl_body!( + $self, + $strategy_ty, + segment, + segment_abs_start, + segment_len, + segment_state, + segment_stats, + $plan, + $buffers, + $collect, + $priceset, + $current, + $opt_state, + $keep_plan, + 'segment, + ) + }; + let pass = &mut $buffers.pass; + pass.reps = end_reps; + pass.litlen = end_litlen; + pass.cursor += consumed_len; + } + }}; +} + /// `collect_optimal_candidates_initialized` body parameterized over the per-CPU /// kernel: the `$cpl` path is the kernel's `common_prefix_len_ptr` (used in /// the HC chain walk fallback), and the four method-name substitutions @@ -1041,9 +1164,10 @@ macro_rules! collect_optimal_candidates_initialized_body { $abs_pos:ident, $current_abs_end:ident, $sufficient_match_len:ident, - $query:ident, + $reps:ident, + $ll0:ident, $out:ident, - $bt_insert_step:ident, + $bt_insert_range:ident, $cpl:path, $cmf:path $(,)? ) => {{ @@ -1064,59 +1188,37 @@ macro_rules! collect_optimal_candidates_initialized_body { ); debug_assert!(!$self.table.chain_table().is_empty()); let min_match_len = HC_OPT_MIN_MATCH_LEN; - let reps = $query.reps; - let lit_len = $query.lit_len; - let ldm_candidate = $query.ldm_candidate; $out.clear(); if $abs_pos < $self.table.skip_insert_until_abs { - if let Some(ldm) = ldm_candidate { - let mut best_len_for_skip = 0usize; - let _ = crate::encoding::bt::BtMatcher::push_candidate_ladder( - $out, - &mut best_len_for_skip, - ldm, - min_match_len, - ); - } return; } { - // BT tree catch-up folded inline (was a per-position call to - // bt_update_tree_until): insert the positions the parser skipped into - // the binary tree before this position's search. Upstream zstd - // ZSTD_updateTree shape; `$bt_insert_step` overshoots the target with - // no clamp (forward skips match-covered positions), exactly as C. - // SAFETY: caller is in the same target_feature umbrella as - // `$bt_insert_step`; the runtime kernel detector already gated entry. + // BT tree catch-up: insert the positions the parser skipped into the + // binary tree before this position's search, as one run (upstream + // zstd ZSTD_updateTree). `$bt_insert_range` overshoots the target + // with no clamp (a long match skips the positions it covers), + // exactly as upstream. if $self.table.skip_insert_until_abs < $self.table.history_abs_start { $self.table.skip_insert_until_abs = $self.table.history_abs_start; } - let mut update_abs = $self.table.skip_insert_until_abs; + let update_abs = $self.table.skip_insert_until_abs; // No rebase guard: `arm_block_positions` cleared every position in // this block before the parse began, and the catch-up only inserts // positions below `$abs_pos`, which is in it. debug_assert!($self.table.can_skip_rebase_check($abs_pos)); - while update_abs < $abs_pos { - let forward = unsafe { + if update_abs < $abs_pos { + // SAFETY: caller is in the same target_feature umbrella as + // `$bt_insert_range`; the runtime kernel detector gated entry. + let _ = unsafe { $self .table - .$bt_insert_step(update_abs, $current_abs_end, $abs_pos) + .$bt_insert_range(update_abs, $abs_pos, $current_abs_end, $abs_pos) }; - update_abs += forward.max(1); } $self.table.skip_insert_until_abs = $abs_pos; } let current_idx = $abs_pos - $self.table.history_abs_start; if current_idx + 4 > $self.table.live_history().len() { - if let Some(ldm) = ldm_candidate { - let mut best_len_for_skip = 0usize; - let _ = crate::encoding::bt::BtMatcher::push_candidate_ladder( - $out, - &mut best_len_for_skip, - ldm, - min_match_len, - ); - } return; } let mut best_len_for_skip = 0usize; @@ -1142,21 +1244,13 @@ macro_rules! collect_optimal_candidates_initialized_body { min_match_len, best_len_ref, $out, - reps, - lit_len, + $reps, + $ll0, use_hash3, $cpl, $cmf, ); } - if let Some(ldm) = ldm_candidate { - let _ = crate::encoding::bt::BtMatcher::push_candidate_ladder( - $out, - &mut best_len_for_skip, - ldm, - min_match_len, - ); - } }}; } impl HcMatchGenerator { @@ -1244,143 +1338,125 @@ impl HcMatchGenerator { debug_assert_eq!(profile.favor_small_offsets, S::FAVOR_SMALL_OFFSETS); let mut opt_state = core::mem::replace(&mut self.backend.bt_mut().opt_state, HcOptState::new()); - opt_state.rescale_freqs(current, profile); + opt_state.rescale_freqs(current, S::ACCURATE_PRICE); let mut best_plan = core::mem::take(&mut self.backend.bt_mut().opt_segment_plan_scratch); best_plan.clear(); - let mut plan_reps = self.table.offset_hist; - let (mut cursor, mut plan_litlen) = - self.table.opt_start_cursor_and_litlen(current_abs_start); - let mut plan_literals_cursor = 0usize; - let match_loop_limit = current_len.saturating_sub(8); + let plan_reps = self.table.offset_hist; + let (cursor, plan_litlen) = self.table.opt_start_cursor_and_litlen(current_abs_start); // Frame-constant LDM presence, resolved once per block (not per segment // and not in the DP hot loop): drives the HAS_LDM const-generic dispatch. let has_ldm = !self.backend.bt_mut().ldm_sequences.is_empty(); - // Resolve the SIMD tier ONCE here, never per segment. The per-literal - // hot loop then runs under a single kernel-monomorphized expansion - // (calling build_optimal_plan_impl_ directly) instead of - // hitting select_kernel()'s OnceLock atomic + a CPU-tier match on every - // build_optimal_plan call. Mirrors the Fast matcher dispatch shape. + // Resolve the SIMD tier ONCE here: the whole block then runs under one + // kernel-monomorphized pass (`run_optimal_block_`) instead of + // hitting select_kernel()'s OnceLock atomic + a CPU-tier match per + // segment. Mirrors the Fast matcher dispatch shape. macro_rules! run_main_loop { - ($impl_wrapper:ident) => {{ - while cursor < match_loop_limit { - let remaining_len = current_len - cursor; - let segment_abs_start = current_abs_start + cursor; - let segment_start = best_plan.len(); - let state = HcOptimalPlanState { - block_offset: cursor, - reps: plan_reps, - litlen: plan_litlen, - profile, - }; - let (_, end_reps, end_litlen, consumed_len) = - match (S::ACCURATE_PRICE, S::FAVOR_SMALL_OFFSETS, has_ldm) { - (true, false, false) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut best_plan, - &mut plan_buffers, - ) - }, - (true, false, true) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut best_plan, - &mut plan_buffers, - ) - }, - (true, true, false) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut best_plan, - &mut plan_buffers, - ) - }, - (true, true, true) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut best_plan, - &mut plan_buffers, - ) - }, - (false, false, false) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut best_plan, - &mut plan_buffers, - ) - }, - (false, false, true) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut best_plan, - &mut plan_buffers, - ) - }, - (false, true, false) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut best_plan, - &mut plan_buffers, - ) - }, - (false, true, true) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut best_plan, - &mut plan_buffers, - ) - }, - }; - // On a no-match segment (the per-literal case that dominates - // near-random input) nothing was emitted, so the stats - // update is a guaranteed no-op (it early-returns on an empty - // plan slice). Skip the per-literal call + its marshalling. - if best_plan.len() > segment_start { - BtMatcher::update_plan_stats_segment( + ($block:ident) => {{ + match (S::ACCURATE_PRICE, S::FAVOR_SMALL_OFFSETS, has_ldm) { + (true, false, false) => unsafe { + self.$block::( current, - current_len, - &best_plan[segment_start..], - &mut plan_literals_cursor, - &mut plan_reps, + current_abs_start, + cursor, + plan_litlen, + plan_reps, + profile, &mut opt_state, - profile.accurate, - ); - } - plan_reps = end_reps; - plan_litlen = end_litlen; - cursor += consumed_len; + &mut best_plan, + &mut plan_buffers, + ) + }, + (true, false, true) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + plan_litlen, + plan_reps, + profile, + &mut opt_state, + &mut best_plan, + &mut plan_buffers, + ) + }, + (true, true, false) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + plan_litlen, + plan_reps, + profile, + &mut opt_state, + &mut best_plan, + &mut plan_buffers, + ) + }, + (true, true, true) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + plan_litlen, + plan_reps, + profile, + &mut opt_state, + &mut best_plan, + &mut plan_buffers, + ) + }, + (false, false, false) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + plan_litlen, + plan_reps, + profile, + &mut opt_state, + &mut best_plan, + &mut plan_buffers, + ) + }, + (false, false, true) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + plan_litlen, + plan_reps, + profile, + &mut opt_state, + &mut best_plan, + &mut plan_buffers, + ) + }, + (false, true, false) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + plan_litlen, + plan_reps, + profile, + &mut opt_state, + &mut best_plan, + &mut plan_buffers, + ) + }, + (false, true, true) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + plan_litlen, + plan_reps, + profile, + &mut opt_state, + &mut best_plan, + &mut plan_buffers, + ) + }, } }}; } @@ -1390,19 +1466,19 @@ impl HcMatchGenerator { feature = "kernel-neon" ))] unsafe { - run_main_loop!(build_optimal_plan_impl_neon); + run_main_loop!(run_optimal_block_neon); } #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] { use crate::encoding::fastpath::FastpathKernel; match self.table.kernel { #[cfg(feature = "kernel-avx2")] - FastpathKernel::Avx2Bmi2 => run_main_loop!(build_optimal_plan_impl_avx2_bmi2), + FastpathKernel::Avx2Bmi2 => run_main_loop!(run_optimal_block_avx2_bmi2), #[cfg(feature = "kernel-sse")] - FastpathKernel::Sse2 => run_main_loop!(build_optimal_plan_impl_sse2), + FastpathKernel::Sse2 => run_main_loop!(run_optimal_block_sse2), #[cfg(feature = "kernel-sse")] - FastpathKernel::Sse42 => run_main_loop!(build_optimal_plan_impl_sse42), - FastpathKernel::Scalar => run_main_loop!(build_optimal_plan_impl_scalar), + FastpathKernel::Sse42 => run_main_loop!(run_optimal_block_sse42), + FastpathKernel::Scalar => run_main_loop!(run_optimal_block_scalar), } } #[cfg(all( @@ -1411,7 +1487,7 @@ impl HcMatchGenerator { feature = "kernel-simd128" ))] unsafe { - run_main_loop!(build_optimal_plan_impl_simd128); + run_main_loop!(run_optimal_block_simd128); } #[cfg(not(any( all( @@ -1428,7 +1504,7 @@ impl HcMatchGenerator { ) )))] { - run_main_loop!(build_optimal_plan_impl_scalar); + run_main_loop!(run_optimal_block_scalar); } self.table @@ -1466,138 +1542,122 @@ impl HcMatchGenerator { debug_assert_eq!(seed_profile.favor_small_offsets, S::FAVOR_SMALL_OFFSETS); let mut opt_state = core::mem::replace(&mut self.backend.bt_mut().opt_state, HcOptState::new()); - opt_state.rescale_freqs(current, seed_profile); - let mut seed_reps = self.table.offset_hist; - let (mut cursor, mut seed_litlen) = - self.table.opt_start_cursor_and_litlen(current_abs_start); - let mut seed_literals_cursor = 0usize; + opt_state.rescale_freqs(current, S::ACCURATE_PRICE); + let seed_reps = self.table.offset_hist; + let (cursor, seed_litlen) = self.table.opt_start_cursor_and_litlen(current_abs_start); let mut seed_plan = core::mem::take(&mut self.backend.bt_mut().opt_seed_plan_scratch); seed_plan.clear(); - let match_loop_limit = current_len.saturating_sub(8); let has_ldm = !self.backend.bt_mut().ldm_sequences.is_empty(); - // SIMD tier resolved ONCE (see start_matching_optimal): the per-literal - // seed loop runs under a single kernel-monomorphized expansion, never - // re-entering select_kernel() per segment. + // SIMD tier resolved ONCE (see start_matching_optimal): the seed pass + // runs as one kernel-monomorphized block pass. It keeps the statistics + // and drops the plan (`KEEP_PLAN = false`). macro_rules! run_seed_loop { - ($impl_wrapper:ident) => {{ - while cursor < match_loop_limit { - let remaining_len = current_len - cursor; - let segment_abs_start = current_abs_start + cursor; - let segment_start = seed_plan.len(); - let state = HcOptimalPlanState { - block_offset: cursor, - reps: seed_reps, - litlen: seed_litlen, - profile: seed_profile, - }; - let (_, end_reps, end_litlen, consumed_len) = - match (S::ACCURATE_PRICE, S::FAVOR_SMALL_OFFSETS, has_ldm) { - (true, false, false) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut seed_plan, - &mut *plan_buffers, - ) - }, - (true, false, true) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut seed_plan, - &mut *plan_buffers, - ) - }, - (true, true, false) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut seed_plan, - &mut *plan_buffers, - ) - }, - (true, true, true) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut seed_plan, - &mut *plan_buffers, - ) - }, - (false, false, false) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut seed_plan, - &mut *plan_buffers, - ) - }, - (false, false, true) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut seed_plan, - &mut *plan_buffers, - ) - }, - (false, true, false) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut seed_plan, - &mut *plan_buffers, - ) - }, - (false, true, true) => unsafe { - self.$impl_wrapper::( - ¤t[cursor..], - segment_abs_start, - remaining_len, - state, - &opt_state, - &mut seed_plan, - &mut *plan_buffers, - ) - }, - }; - // No-match segment: stats update no-ops on the empty slice - // and the truncate has nothing to drop; skip both. - if seed_plan.len() > segment_start { - BtMatcher::update_plan_stats_segment( + ($block:ident) => {{ + match (S::ACCURATE_PRICE, S::FAVOR_SMALL_OFFSETS, has_ldm) { + (true, false, false) => unsafe { + self.$block::( current, - current_len, - &seed_plan[segment_start..], - &mut seed_literals_cursor, - &mut seed_reps, + current_abs_start, + cursor, + seed_litlen, + seed_reps, + seed_profile, &mut opt_state, - seed_profile.accurate, - ); - seed_plan.truncate(segment_start); - } - seed_reps = end_reps; - seed_litlen = end_litlen; - cursor += consumed_len; + &mut seed_plan, + &mut *plan_buffers, + ) + }, + (true, false, true) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + seed_litlen, + seed_reps, + seed_profile, + &mut opt_state, + &mut seed_plan, + &mut *plan_buffers, + ) + }, + (true, true, false) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + seed_litlen, + seed_reps, + seed_profile, + &mut opt_state, + &mut seed_plan, + &mut *plan_buffers, + ) + }, + (true, true, true) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + seed_litlen, + seed_reps, + seed_profile, + &mut opt_state, + &mut seed_plan, + &mut *plan_buffers, + ) + }, + (false, false, false) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + seed_litlen, + seed_reps, + seed_profile, + &mut opt_state, + &mut seed_plan, + &mut *plan_buffers, + ) + }, + (false, false, true) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + seed_litlen, + seed_reps, + seed_profile, + &mut opt_state, + &mut seed_plan, + &mut *plan_buffers, + ) + }, + (false, true, false) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + seed_litlen, + seed_reps, + seed_profile, + &mut opt_state, + &mut seed_plan, + &mut *plan_buffers, + ) + }, + (false, true, true) => unsafe { + self.$block::( + current, + current_abs_start, + cursor, + seed_litlen, + seed_reps, + seed_profile, + &mut opt_state, + &mut seed_plan, + &mut *plan_buffers, + ) + }, } }}; } @@ -1607,19 +1667,19 @@ impl HcMatchGenerator { feature = "kernel-neon" ))] unsafe { - run_seed_loop!(build_optimal_plan_impl_neon); + run_seed_loop!(run_optimal_block_neon); } #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] { use crate::encoding::fastpath::FastpathKernel; match self.table.kernel { #[cfg(feature = "kernel-avx2")] - FastpathKernel::Avx2Bmi2 => run_seed_loop!(build_optimal_plan_impl_avx2_bmi2), + FastpathKernel::Avx2Bmi2 => run_seed_loop!(run_optimal_block_avx2_bmi2), #[cfg(feature = "kernel-sse")] - FastpathKernel::Sse2 => run_seed_loop!(build_optimal_plan_impl_sse2), + FastpathKernel::Sse2 => run_seed_loop!(run_optimal_block_sse2), #[cfg(feature = "kernel-sse")] - FastpathKernel::Sse42 => run_seed_loop!(build_optimal_plan_impl_sse42), - FastpathKernel::Scalar => run_seed_loop!(build_optimal_plan_impl_scalar), + FastpathKernel::Sse42 => run_seed_loop!(run_optimal_block_sse42), + FastpathKernel::Scalar => run_seed_loop!(run_optimal_block_scalar), } } #[cfg(all( @@ -1628,7 +1688,7 @@ impl HcMatchGenerator { feature = "kernel-simd128" ))] unsafe { - run_seed_loop!(build_optimal_plan_impl_simd128); + run_seed_loop!(run_optimal_block_simd128); } #[cfg(not(any( all( @@ -1645,7 +1705,7 @@ impl HcMatchGenerator { ) )))] { - run_seed_loop!(build_optimal_plan_impl_scalar); + run_seed_loop!(run_optimal_block_scalar); } seed_plan.clear(); self.backend.bt_mut().opt_seed_plan_scratch = seed_plan; @@ -1710,12 +1770,13 @@ impl HcMatchGenerator { // Nothing in the buffer answers a query yet: the block that filled // it is over, and the parser is about to start another. candidates_searched_at: None, + pass: HcBlockPass::default(), } } - /// NEON-umbrella DP body. Inlines - /// `collect_optimal_candidates_initialized_neon` (and its entire - /// per-position pipeline) directly into the DP loop. + /// NEON-umbrella pass over one block (see [`optimal_block_body!`]). + /// `collect_optimal_candidates_initialized_neon` shares the umbrella, so + /// the per-position pipeline needs no feature re-dispatch. #[cfg(all( target_arch = "aarch64", target_endian = "little", @@ -1723,33 +1784,39 @@ impl HcMatchGenerator { ))] #[target_feature(enable = "neon")] #[allow(clippy::too_many_arguments)] - unsafe fn build_optimal_plan_impl_neon< + unsafe fn run_optimal_block_neon< S: crate::encoding::strategy::Strategy, const ACCURATE_PRICE: bool, const FAVOR_SMALL_OFFSETS: bool, const HAS_LDM: bool, + const KEEP_PLAN: bool, >( &mut self, current: &[u8], current_abs_start: usize, - current_len: usize, - initial_state: HcOptimalPlanState, - stats: &HcOptState, - out: &mut Vec, + cursor: usize, + litlen: usize, + reps: [u32; 3], + profile: HcOptimalCostProfile, + opt_state: &mut HcOptState, + plan: &mut Vec, buffers: &mut HcOptimalPlanBuffers, - ) -> (u32, [u32; 3], usize, usize) { - build_optimal_plan_impl_body!( + ) { + optimal_block_body!( self, S, current, current_abs_start, - current_len, - initial_state, - stats, - out, + cursor, + litlen, + reps, + profile, + opt_state, + plan, buffers, + KEEP_PLAN, collect_optimal_candidates_initialized_neon, - crate::encoding::hc::priceset::priceset_range_nonabort_neon, + crate::encoding::hc::priceset::priceset_range_nonabort_neon::, ) } @@ -1762,37 +1829,43 @@ impl HcMatchGenerator { ))] #[target_feature(enable = "sse4.2")] #[allow(clippy::too_many_arguments)] - unsafe fn build_optimal_plan_impl_sse42< + unsafe fn run_optimal_block_sse42< S: crate::encoding::strategy::Strategy, const ACCURATE_PRICE: bool, const FAVOR_SMALL_OFFSETS: bool, const HAS_LDM: bool, + const KEEP_PLAN: bool, >( &mut self, current: &[u8], current_abs_start: usize, - current_len: usize, - initial_state: HcOptimalPlanState, - stats: &HcOptState, - out: &mut Vec, + cursor: usize, + litlen: usize, + reps: [u32; 3], + profile: HcOptimalCostProfile, + opt_state: &mut HcOptState, + plan: &mut Vec, buffers: &mut HcOptimalPlanBuffers, - ) -> (u32, [u32; 3], usize, usize) { - build_optimal_plan_impl_body!( + ) { + optimal_block_body!( self, S, current, current_abs_start, - current_len, - initial_state, - stats, - out, + cursor, + litlen, + reps, + profile, + opt_state, + plan, buffers, + KEEP_PLAN, collect_optimal_candidates_initialized_sse42, - crate::encoding::hc::priceset::priceset_range_nonabort_sse41, + crate::encoding::hc::priceset::priceset_range_nonabort_sse41::, ) } - /// SSE2 twin of [`Self::build_optimal_plan_impl_sse42`] for x86 CPUs + /// SSE2 twin of [`Self::run_optimal_block_sse42`] for x86 CPUs /// without SSE4.1/4.2: same 128-bit pipeline, with the price set using /// the SSE2 unsigned-compare emulation. #[cfg(all( @@ -1801,33 +1874,39 @@ impl HcMatchGenerator { ))] #[target_feature(enable = "sse2")] #[allow(clippy::too_many_arguments)] - unsafe fn build_optimal_plan_impl_sse2< + unsafe fn run_optimal_block_sse2< S: crate::encoding::strategy::Strategy, const ACCURATE_PRICE: bool, const FAVOR_SMALL_OFFSETS: bool, const HAS_LDM: bool, + const KEEP_PLAN: bool, >( &mut self, current: &[u8], current_abs_start: usize, - current_len: usize, - initial_state: HcOptimalPlanState, - stats: &HcOptState, - out: &mut Vec, + cursor: usize, + litlen: usize, + reps: [u32; 3], + profile: HcOptimalCostProfile, + opt_state: &mut HcOptState, + plan: &mut Vec, buffers: &mut HcOptimalPlanBuffers, - ) -> (u32, [u32; 3], usize, usize) { - build_optimal_plan_impl_body!( + ) { + optimal_block_body!( self, S, current, current_abs_start, - current_len, - initial_state, - stats, - out, + cursor, + litlen, + reps, + profile, + opt_state, + plan, buffers, + KEEP_PLAN, collect_optimal_candidates_initialized_sse2, - crate::encoding::hc::priceset::priceset_range_nonabort_sse2, + crate::encoding::hc::priceset::priceset_range_nonabort_sse2::, ) } @@ -1837,33 +1916,39 @@ impl HcMatchGenerator { ))] #[target_feature(enable = "avx2,bmi2")] #[allow(clippy::too_many_arguments)] - unsafe fn build_optimal_plan_impl_avx2_bmi2< + unsafe fn run_optimal_block_avx2_bmi2< S: crate::encoding::strategy::Strategy, const ACCURATE_PRICE: bool, const FAVOR_SMALL_OFFSETS: bool, const HAS_LDM: bool, + const KEEP_PLAN: bool, >( &mut self, current: &[u8], current_abs_start: usize, - current_len: usize, - initial_state: HcOptimalPlanState, - stats: &HcOptState, - out: &mut Vec, + cursor: usize, + litlen: usize, + reps: [u32; 3], + profile: HcOptimalCostProfile, + opt_state: &mut HcOptState, + plan: &mut Vec, buffers: &mut HcOptimalPlanBuffers, - ) -> (u32, [u32; 3], usize, usize) { - build_optimal_plan_impl_body!( + ) { + optimal_block_body!( self, S, current, current_abs_start, - current_len, - initial_state, - stats, - out, + cursor, + litlen, + reps, + profile, + opt_state, + plan, buffers, + KEEP_PLAN, collect_optimal_candidates_initialized_avx2_bmi2, - crate::encoding::hc::priceset::priceset_range_nonabort_avx2, + crate::encoding::hc::priceset::priceset_range_nonabort_avx2::, ) } @@ -1888,33 +1973,39 @@ impl HcMatchGenerator { allow(dead_code) )] #[allow(clippy::too_many_arguments)] - fn build_optimal_plan_impl_scalar< + fn run_optimal_block_scalar< S: crate::encoding::strategy::Strategy, const ACCURATE_PRICE: bool, const FAVOR_SMALL_OFFSETS: bool, const HAS_LDM: bool, + const KEEP_PLAN: bool, >( &mut self, current: &[u8], current_abs_start: usize, - current_len: usize, - initial_state: HcOptimalPlanState, - stats: &HcOptState, - out: &mut Vec, + cursor: usize, + litlen: usize, + reps: [u32; 3], + profile: HcOptimalCostProfile, + opt_state: &mut HcOptState, + plan: &mut Vec, buffers: &mut HcOptimalPlanBuffers, - ) -> (u32, [u32; 3], usize, usize) { - build_optimal_plan_impl_body!( + ) { + optimal_block_body!( self, S, current, current_abs_start, - current_len, - initial_state, - stats, - out, + cursor, + litlen, + reps, + profile, + opt_state, + plan, buffers, + KEEP_PLAN, collect_optimal_candidates_initialized_scalar, - crate::encoding::hc::priceset::priceset_range_nonabort_scalar, + crate::encoding::hc::priceset::priceset_range_nonabort_scalar::, ) } @@ -1931,33 +2022,39 @@ impl HcMatchGenerator { // target_feature fn. #[allow(unused_unsafe)] #[allow(clippy::too_many_arguments)] - unsafe fn build_optimal_plan_impl_simd128< + unsafe fn run_optimal_block_simd128< S: crate::encoding::strategy::Strategy, const ACCURATE_PRICE: bool, const FAVOR_SMALL_OFFSETS: bool, const HAS_LDM: bool, + const KEEP_PLAN: bool, >( &mut self, current: &[u8], current_abs_start: usize, - current_len: usize, - initial_state: HcOptimalPlanState, - stats: &HcOptState, - out: &mut Vec, + cursor: usize, + litlen: usize, + reps: [u32; 3], + profile: HcOptimalCostProfile, + opt_state: &mut HcOptState, + plan: &mut Vec, buffers: &mut HcOptimalPlanBuffers, - ) -> (u32, [u32; 3], usize, usize) { - build_optimal_plan_impl_body!( + ) { + optimal_block_body!( self, S, current, current_abs_start, - current_len, - initial_state, - stats, - out, + cursor, + litlen, + reps, + profile, + opt_state, + plan, buffers, + KEEP_PLAN, collect_optimal_candidates_initialized_simd128, - crate::encoding::hc::priceset::priceset_range_nonabort_simd128, + crate::encoding::hc::priceset::priceset_range_nonabort_simd128::, ) } @@ -1972,6 +2069,9 @@ impl HcMatchGenerator { ) { use crate::encoding::strategy::{self, StrategyTag}; self.table.ensure_tables(); + self.table.capture_block_coords(); + let reps = &query.reps; + let ll0 = query.lit_len == 0; // Dispatch purely from `self.strategy_tag` (set by // `configure()`). Tests must configure the matcher the same // way production does — wiring up `table.hash3_log` directly @@ -1983,7 +2083,8 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ), StrategyTag::BtUltra => self @@ -1991,7 +2092,8 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ), StrategyTag::Btlazy2 => self @@ -1999,14 +2101,16 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ), StrategyTag::BtOpt => self.collect_optimal_candidates_initialized::( abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ), StrategyTag::Fast | StrategyTag::Dfast | StrategyTag::Greedy | StrategyTag::Lazy => { @@ -2022,6 +2126,9 @@ impl HcMatchGenerator { ) } } + if let Some(ldm) = query.ldm_candidate { + BtMatcher::push_ldm_candidate(out, ldm, HC_OPT_MIN_MATCH_LEN); + } } /// Cross-platform entry. Picks the kernel-specific variant so the per- @@ -2029,7 +2136,7 @@ impl HcMatchGenerator { /// collect / HC chain walk) runs inside a single `target_feature` /// umbrella — all inner SIMD probes inline without ABI barriers. /// - /// The on-encode hot path bypasses this dispatcher: `build_optimal_plan_impl_` + /// The on-encode hot path bypasses this dispatcher: `run_optimal_block_` /// calls the matching `_` variant directly. This entry is kept /// for the cfg(test)-only `collect_optimal_candidates` shim and any /// future caller that isn't already inside a kernel umbrella. @@ -2050,7 +2157,8 @@ impl HcMatchGenerator { abs_pos: usize, current_abs_end: usize, sufficient_match_len: usize, - query: HcCandidateQuery, + reps: &[u32; 3], + ll0: bool, out: &mut Vec, ) { #[cfg(all( @@ -2063,7 +2171,8 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ) } @@ -2077,7 +2186,8 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ) }, @@ -2087,7 +2197,8 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ) }, @@ -2097,7 +2208,8 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ) }, @@ -2105,7 +2217,8 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ), } @@ -2124,7 +2237,8 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ) } @@ -2147,17 +2261,17 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, ) } } - /// NEON-umbrella variant. Every inner helper (`bt_update_tree_until_neon`, - /// `for_each_repcode_candidate_with_reps_neon`, `hash3_candidate_neon`, - /// `bt_insert_and_collect_matches_neon`, `fastpath::neon:: - /// common_prefix_len_ptr`) shares the NEON umbrella so the per-position - /// pipeline executes as a single straight-line inline sequence. + /// NEON-umbrella variant: the repeat and hash3 probes, the tree walk and + /// `fastpath::neon::common_prefix_len_ptr` share the NEON umbrella so the + /// per-position search runs as one straight-line sequence, and the tree + /// catch-up (`bt_insert_range_neon`) runs under the same tier. #[cfg(all( target_arch = "aarch64", target_endian = "little", @@ -2172,7 +2286,8 @@ impl HcMatchGenerator { abs_pos: usize, current_abs_end: usize, sufficient_match_len: usize, - query: HcCandidateQuery, + reps: &[u32; 3], + ll0: bool, out: &mut Vec, ) { collect_optimal_candidates_initialized_body!( @@ -2181,9 +2296,10 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, - bt_insert_step_no_rebase_neon, + bt_insert_range_neon, crate::encoding::fastpath::neon::common_prefix_len_ptr, crate::encoding::fastpath::neon::count_match_from_indices, ) @@ -2202,7 +2318,8 @@ impl HcMatchGenerator { abs_pos: usize, current_abs_end: usize, sufficient_match_len: usize, - query: HcCandidateQuery, + reps: &[u32; 3], + ll0: bool, out: &mut Vec, ) { collect_optimal_candidates_initialized_body!( @@ -2211,9 +2328,10 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, - bt_insert_step_no_rebase_sse2, + bt_insert_range_sse2, crate::encoding::fastpath::sse2::common_prefix_len_ptr, crate::encoding::fastpath::sse2::count_match_from_indices, ) @@ -2238,7 +2356,8 @@ impl HcMatchGenerator { abs_pos: usize, current_abs_end: usize, sufficient_match_len: usize, - query: HcCandidateQuery, + reps: &[u32; 3], + ll0: bool, out: &mut Vec, ) { collect_optimal_candidates_initialized_body!( @@ -2247,9 +2366,10 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, - bt_insert_step_no_rebase_sse2, + bt_insert_range_sse2, crate::encoding::fastpath::sse2::common_prefix_len_ptr, crate::encoding::fastpath::sse2::count_match_from_indices, ) @@ -2268,7 +2388,8 @@ impl HcMatchGenerator { abs_pos: usize, current_abs_end: usize, sufficient_match_len: usize, - query: HcCandidateQuery, + reps: &[u32; 3], + ll0: bool, out: &mut Vec, ) { collect_optimal_candidates_initialized_body!( @@ -2277,9 +2398,10 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, - bt_insert_step_no_rebase_avx2_bmi2, + bt_insert_range_avx2_bmi2, crate::encoding::fastpath::avx2_bmi2::common_prefix_len_ptr, crate::encoding::fastpath::avx2_bmi2::count_match_from_indices, ) @@ -2305,7 +2427,8 @@ impl HcMatchGenerator { abs_pos: usize, current_abs_end: usize, sufficient_match_len: usize, - query: HcCandidateQuery, + reps: &[u32; 3], + ll0: bool, out: &mut Vec, ) { collect_optimal_candidates_initialized_body!( @@ -2314,9 +2437,10 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, - bt_insert_step_no_rebase_simd128, + bt_insert_range_simd128, crate::encoding::fastpath::simd128::common_prefix_len_ptr, crate::encoding::fastpath::simd128::count_match_from_indices, ) @@ -2338,7 +2462,8 @@ impl HcMatchGenerator { abs_pos: usize, current_abs_end: usize, sufficient_match_len: usize, - query: HcCandidateQuery, + reps: &[u32; 3], + ll0: bool, out: &mut Vec, ) { collect_optimal_candidates_initialized_body!( @@ -2347,9 +2472,10 @@ impl HcMatchGenerator { abs_pos, current_abs_end, sufficient_match_len, - query, + reps, + ll0, out, - bt_insert_step_no_rebase_scalar, + bt_insert_range_scalar, crate::encoding::fastpath::scalar::common_prefix_len_ptr, crate::encoding::fastpath::scalar::count_match_from_indices, ) diff --git a/zstd/src/encoding/hc/priceset.rs b/zstd/src/encoding/hc/priceset.rs index eeac395ca..0e29ddada 100644 --- a/zstd/src/encoding/hc/priceset.rs +++ b/zstd/src/encoding/hc/priceset.rs @@ -51,7 +51,7 @@ unsafe fn priceset_improved_mask8_avx2(next_cost: &[u32; 8], node_price: &[u32]) /// so the SoA vector path stays byte-identical. #[inline(always)] #[allow(clippy::too_many_arguments)] -fn priceset_next_cost( +fn priceset_next_cost( profile: HcOptimalCostProfile, stats: &HcOptState, ml_cache: &mut [[u32; 2]], @@ -61,8 +61,9 @@ fn priceset_next_cost( off_price: u32, base_cost: u32, ) -> u32 { - let ml_price = - BtMatcher::cached_match_length_price(profile, stats, match_len, ml_cache, ml_stamp); + let ml_price = BtMatcher::cached_match_length_price::( + profile, stats, match_len, ml_cache, ml_stamp, + ); let seq_cost = BtMatcher::add_prices( ll0_price, profile.match_price_from_parts(off_price, ml_price, stats), @@ -94,7 +95,7 @@ fn priceset_next_cost( ), allow(dead_code) )] -pub(crate) fn priceset_range_nonabort_scalar( +pub(crate) fn priceset_range_nonabort_scalar( node_prices: &mut [u32], nodes: &mut [MaybeUninit], ml_cache: &mut [[u32; 2]], @@ -108,23 +109,18 @@ pub(crate) fn priceset_range_nonabort_scalar( off_price: u32, base_cost: u32, off: u32, - reps: [u32; 3], last_pos: usize, ) -> usize { let mut new_last = last_pos; for ml in start..=max { - let next_cost = priceset_next_cost( + let next_cost = priceset_next_cost::( profile, stats, ml_cache, ml_stamp, ml, ll0_price, off_price, base_cost, ); let next = pos + ml; if next_cost < node_prices[next] { node_prices[next] = next_cost; - nodes[next].write(HcOptimalNode { - off, - mlen: ml as u32, - litlen: 0, - reps, - }); + // SAFETY: `nodes[next]` is an in-bounds cell of the slice. + unsafe { HcOptimalNode::write_match_end(nodes[next].as_mut_ptr(), off, ml as u32) }; if next > new_last { new_last = next; } @@ -162,7 +158,7 @@ pub(crate) fn priceset_range_nonabort_scalar( )), allow(dead_code) )] -fn priceset_range_vec( +fn priceset_range_vec( node_prices: &mut [u32], nodes: &mut [MaybeUninit], ml_cache: &mut [[u32; 2]], @@ -176,7 +172,6 @@ fn priceset_range_vec( off_price: u32, base_cost: u32, off: u32, - reps: [u32; 3], last_pos: usize, deint: impl Fn(&[[u32; 2]], u32) -> Option<[u32; W]>, mask: impl Fn(&[u32; W], &[u32]) -> u8, @@ -215,7 +210,7 @@ fn priceset_range_vec( } } else { for (k, slot) in buf.iter_mut().enumerate() { - *slot = priceset_next_cost( + *slot = priceset_next_cost::( profile, stats, ml_cache, @@ -234,12 +229,10 @@ fn priceset_range_vec( bits &= bits - 1; let next = base_next + k; node_prices[next] = buf[k]; - nodes[next].write(HcOptimalNode { - off, - mlen: (ml + k) as u32, - litlen: 0, - reps, - }); + // SAFETY: `nodes[next]` is an in-bounds cell of the slice. + unsafe { + HcOptimalNode::write_match_end(nodes[next].as_mut_ptr(), off, (ml + k) as u32) + }; if next > new_last { new_last = next; } @@ -247,18 +240,14 @@ fn priceset_range_vec( ml += W; } while ml <= max { - let next_cost = priceset_next_cost( + let next_cost = priceset_next_cost::( profile, stats, ml_cache, ml_stamp, ml, ll0_price, off_price, base_cost, ); let next = pos + ml; if next_cost < node_prices[next] { node_prices[next] = next_cost; - nodes[next].write(HcOptimalNode { - off, - mlen: ml as u32, - litlen: 0, - reps, - }); + // SAFETY: `nodes[next]` is an in-bounds cell of the slice. + unsafe { HcOptimalNode::write_match_end(nodes[next].as_mut_ptr(), off, ml as u32) }; if next > new_last { new_last = next; } @@ -322,7 +311,7 @@ unsafe fn priceset_cached_prices8_avx2(cells: &[[u32; 2]], stamp: u32) -> Option #[target_feature(enable = "avx2")] #[inline] #[allow(clippy::too_many_arguments)] -pub(crate) unsafe fn priceset_range_nonabort_avx2( +pub(crate) unsafe fn priceset_range_nonabort_avx2( node_prices: &mut [u32], nodes: &mut [MaybeUninit], ml_cache: &mut [[u32; 2]], @@ -336,10 +325,9 @@ pub(crate) unsafe fn priceset_range_nonabort_avx2( off_price: u32, base_cost: u32, off: u32, - reps: [u32; 3], last_pos: usize, ) -> usize { - priceset_range_vec::<8>( + priceset_range_vec::<8, ACCURATE>( node_prices, nodes, ml_cache, @@ -353,7 +341,6 @@ pub(crate) unsafe fn priceset_range_nonabort_avx2( off_price, base_cost, off, - reps, last_pos, // SAFETY: both closures run inside this fn's avx2 target_feature umbrella. |cells, stamp| unsafe { priceset_cached_prices8_avx2(cells, stamp) }, @@ -405,7 +392,7 @@ unsafe fn priceset_improved_mask4_neon(next_cost: &[u32; 4], node_price: &[u32]) #[target_feature(enable = "neon")] #[inline] #[allow(clippy::too_many_arguments)] -pub(crate) unsafe fn priceset_range_nonabort_neon( +pub(crate) unsafe fn priceset_range_nonabort_neon( node_prices: &mut [u32], nodes: &mut [MaybeUninit], ml_cache: &mut [[u32; 2]], @@ -419,10 +406,9 @@ pub(crate) unsafe fn priceset_range_nonabort_neon( off_price: u32, base_cost: u32, off: u32, - reps: [u32; 3], last_pos: usize, ) -> usize { - priceset_range_vec::<4>( + priceset_range_vec::<4, ACCURATE>( node_prices, nodes, ml_cache, @@ -436,7 +422,6 @@ pub(crate) unsafe fn priceset_range_nonabort_neon( off_price, base_cost, off, - reps, last_pos, // SAFETY: both closures run inside this fn's neon target_feature umbrella. |cells, stamp| unsafe { priceset_cached_prices4_neon(cells, stamp) }, @@ -537,7 +522,7 @@ unsafe fn priceset_improved_mask4_sse41(next_cost: &[u32; 4], node_price: &[u32] #[target_feature(enable = "sse4.2")] #[inline] #[allow(clippy::too_many_arguments)] -pub(crate) unsafe fn priceset_range_nonabort_sse41( +pub(crate) unsafe fn priceset_range_nonabort_sse41( node_prices: &mut [u32], nodes: &mut [MaybeUninit], ml_cache: &mut [[u32; 2]], @@ -551,10 +536,9 @@ pub(crate) unsafe fn priceset_range_nonabort_sse41( off_price: u32, base_cost: u32, off: u32, - reps: [u32; 3], last_pos: usize, ) -> usize { - priceset_range_vec::<4>( + priceset_range_vec::<4, ACCURATE>( node_prices, nodes, ml_cache, @@ -568,7 +552,6 @@ pub(crate) unsafe fn priceset_range_nonabort_sse41( off_price, base_cost, off, - reps, last_pos, // SAFETY: both closures run inside this fn's sse4.2 target_feature // umbrella, which covers the SSE2 loader and the SSE4.1 mask. @@ -584,7 +567,7 @@ pub(crate) unsafe fn priceset_range_nonabort_sse41( #[target_feature(enable = "sse2")] #[inline] #[allow(clippy::too_many_arguments)] -pub(crate) unsafe fn priceset_range_nonabort_sse2( +pub(crate) unsafe fn priceset_range_nonabort_sse2( node_prices: &mut [u32], nodes: &mut [MaybeUninit], ml_cache: &mut [[u32; 2]], @@ -598,10 +581,9 @@ pub(crate) unsafe fn priceset_range_nonabort_sse2( off_price: u32, base_cost: u32, off: u32, - reps: [u32; 3], last_pos: usize, ) -> usize { - priceset_range_vec::<4>( + priceset_range_vec::<4, ACCURATE>( node_prices, nodes, ml_cache, @@ -615,7 +597,6 @@ pub(crate) unsafe fn priceset_range_nonabort_sse2( off_price, base_cost, off, - reps, last_pos, // SAFETY: both closures run inside this fn's sse2 target_feature umbrella. |cells, stamp| unsafe { priceset_cached_prices4_sse2(cells, stamp) }, @@ -678,7 +659,7 @@ unsafe fn priceset_improved_mask4_simd128(next_cost: &[u32; 4], node_price: &[u3 #[target_feature(enable = "simd128")] #[inline] #[allow(clippy::too_many_arguments)] -pub(crate) unsafe fn priceset_range_nonabort_simd128( +pub(crate) unsafe fn priceset_range_nonabort_simd128( node_prices: &mut [u32], nodes: &mut [MaybeUninit], ml_cache: &mut [[u32; 2]], @@ -692,10 +673,9 @@ pub(crate) unsafe fn priceset_range_nonabort_simd128( off_price: u32, base_cost: u32, off: u32, - reps: [u32; 3], last_pos: usize, ) -> usize { - priceset_range_vec::<4>( + priceset_range_vec::<4, ACCURATE>( node_prices, nodes, ml_cache, @@ -709,7 +689,6 @@ pub(crate) unsafe fn priceset_range_nonabort_simd128( off_price, base_cost, off, - reps, last_pos, // SAFETY: both closures run inside this fn's simd128 target_feature umbrella. |cells, stamp| unsafe { priceset_cached_prices4_simd128(cells, stamp) }, diff --git a/zstd/src/encoding/levels/fastest.rs b/zstd/src/encoding/levels/fastest.rs index 173df13b4..a3d113183 100644 --- a/zstd/src/encoding/levels/fastest.rs +++ b/zstd/src/encoding/levels/fastest.rs @@ -11,6 +11,7 @@ use crate::{ compression_level_allows_raw_fast_path, }, match_generator::MatchGeneratorDriver, + strategy::StrategyTag, }, }; use alloc::vec::Vec; @@ -168,11 +169,9 @@ pub(crate) fn compress_block_encoded( // consume it: when no sink collects checksums (the common case), and when // the block is headed for the post-split helper, which emits several // physical blocks and records a checksum per partition of its own. + let post_split = post_split_enabled(state.strategy_tag, window_size); #[cfg(all(feature = "lsm", feature = "hash"))] - let post_split_path = rle_byte_opt.is_none() - && !raw_fast_path - && matches!(compression_level, CompressionLevel::Level(16..=22)) - && state.matcher.window_size() >= (1 << 17); + let post_split_path = rle_byte_opt.is_none() && !raw_fast_path && post_split; #[cfg(all(feature = "lsm", feature = "hash"))] let precomputed_checksum = block_checksums .as_ref() @@ -222,9 +221,7 @@ pub(crate) fn compress_block_encoded( } else { // Compress as a standard compressed block uncompressed_data.commit(&mut state.matcher); - if matches!(compression_level, CompressionLevel::Level(16..=22)) - && state.matcher.window_size() >= (1 << 17) - { + if post_split { // This helper may emit multiple physical blocks (compressed or raw) // into `output`; the decompressed-size and (if requested) checksum // sidecars are pushed per physical block from inside the partition @@ -363,8 +360,9 @@ pub(crate) fn compress_block_encoded( /// branch selection and shares the heavy `compress_block` machinery; the /// only differences are how the block is acquired (borrowed slice, no /// copy) and that raw/RLE bodies are emitted straight from `block`. The -/// `Level(16..=22)` post-split branch is unreachable here (the borrowed -/// path is gated to Fast levels), so it is omitted. +/// post-split branch is unreachable here (it needs an optimal-band +/// strategy, and the borrowed path is gated to the fast one), so it is +/// omitted. #[allow(clippy::too_many_arguments)] pub(crate) fn compress_block_encoded_borrowed( state: &mut CompressState, @@ -568,6 +566,26 @@ pub(crate) fn compress_block_encoded_borrowed( } } +/// Whether a compressed block goes through the post-split pass, which may cut +/// it into several blocks along its sequences. Decided by the strategy the +/// frame runs, not by its level, as upstream zstd decides it +/// (`zstd_compress.c`, `ZSTD_resolveBlockSplitterMode`: `strategy >= btopt && +/// windowLog >= 17`), so a parameter set that moves the strategy moves the +/// pass with it. +/// +/// Turning the pass off is not a speed lever. Gated by level it was skipped on +/// the btopt / btultra frames of L13-15, which ran faster there only by paying +/// 6-9% in bytes (z000033[..200000], L13: 90378 against 85017 with the pass). +/// Where the pass costs time, the cost is in the splitter itself, and that is +/// what has to get cheaper. +#[inline] +fn post_split_enabled(strategy_tag: StrategyTag, window_size: u64) -> bool { + matches!( + strategy_tag, + StrategyTag::BtOpt | StrategyTag::BtUltra | StrategyTag::BtUltra2 + ) && window_size >= 1 << 17 +} + /// Whether this block may go out raw without being searched. /// /// The classifier answers from the block's own bytes, which cannot see a repeat diff --git a/zstd/src/encoding/match_generator/tests.rs b/zstd/src/encoding/match_generator/tests.rs index a647c1813..be70eb96d 100644 --- a/zstd/src/encoding/match_generator/tests.rs +++ b/zstd/src/encoding/match_generator/tests.rs @@ -950,26 +950,28 @@ fn btultra2_profile_disables_small_offset_handicap() { !profile.favor_small_offsets, "btultra2 should match upstream zstd opt2 offset pricing" ); - assert!( - profile.accurate, - "btultra2 should use upstream zstd opt2 accurate pricing" - ); + const { + assert!( + ::ACCURATE_PRICE, + "btultra2 should use upstream zstd opt2 accurate pricing" + ); + } } #[test] -fn btultra_profile_keeps_search_depth_budget() { - let p = HcOptimalCostProfile::const_for_strategy::(); +fn btultra_keeps_search_depth_budget() { assert_eq!( - p.max_chain_depth, 64, + ::MAX_CHAIN_DEPTH, + 64, "btultra chain-depth budget must match clevels.h level 18 searchLog 6 (1 << 6 = 64)" ); } #[test] -fn btopt_profile_keeps_search_depth_budget() { - let p = HcOptimalCostProfile::const_for_strategy::(); +fn btopt_keeps_search_depth_budget() { assert_eq!( - p.max_chain_depth, 32, + ::MAX_CHAIN_DEPTH, + 32, "btopt should not cap chain depth below upstream zstd btopt search budget" ); } @@ -1030,7 +1032,7 @@ fn dictionary_entropy_seed_initializes_opt_state_from_tables() { hc.backend.bt_mut().opt_state.rescale_freqs( b"abcd", - HcOptimalCostProfile::const_for_strategy::(), + ::ACCURATE_PRICE, ); let base_ll_freqs: [u32; HC_MAX_LL + 1] = [ @@ -1075,7 +1077,7 @@ fn dictionary_fse_seed_applies_without_huffman_seed() { hc.seed_dictionary_entropy(None, Some(&*ll), Some(&*ml), Some(&*of)); hc.backend.bt_mut().opt_state.rescale_freqs( b"abcd", - HcOptimalCostProfile::const_for_strategy::(), + ::ACCURATE_PRICE, ); let base_ll_freqs: [u32; HC_MAX_LL + 1] = [ @@ -1119,7 +1121,7 @@ fn dictionary_seed_overrides_predef_price_mode_on_tiny_input() { hc.seed_dictionary_entropy(None, Some(&*ll), Some(&*ml), Some(&*of)); hc.backend.bt_mut().opt_state.rescale_freqs( b"abc", - HcOptimalCostProfile::const_for_strategy::(), + ::ACCURATE_PRICE, ); assert!( matches!( @@ -1136,9 +1138,9 @@ fn lit_length_price_blocksize_max_costs_one_extra_bit() { HcOptimalCostProfile::const_for_strategy::(); let mut stats_predef = HcOptState::new(); stats_predef.price_type = HcOptPriceType::Predefined; - let predef_max = profile_predef.lit_length_price(&stats_predef, HC_BLOCKSIZE_MAX); + let predef_max = profile_predef.lit_length_price::(&stats_predef, HC_BLOCKSIZE_MAX); let predef_prev = - profile_predef.lit_length_price(&stats_predef, HC_BLOCKSIZE_MAX.saturating_sub(1)); + profile_predef.lit_length_price::(&stats_predef, HC_BLOCKSIZE_MAX.saturating_sub(1)); assert_eq!( predef_max, predef_prev + HC_BITCOST_MULTIPLIER, @@ -1158,8 +1160,9 @@ fn lit_length_price_blocksize_max_costs_one_extra_bit() { stats_dyn.lit_freq.fill(1); stats_dyn.lit_sum = (HC_MAX_LIT + 1) as u32; stats_dyn.set_base_prices(true); - let dyn_max = profile_dyn.lit_length_price(&stats_dyn, HC_BLOCKSIZE_MAX); - let dyn_prev = profile_dyn.lit_length_price(&stats_dyn, HC_BLOCKSIZE_MAX.saturating_sub(1)); + let dyn_max = profile_dyn.lit_length_price::(&stats_dyn, HC_BLOCKSIZE_MAX); + let dyn_prev = + profile_dyn.lit_length_price::(&stats_dyn, HC_BLOCKSIZE_MAX.saturating_sub(1)); assert_eq!( dyn_max, dyn_prev + HC_BITCOST_MULTIPLIER, @@ -1289,7 +1292,7 @@ fn literal_price_uses_eight_bits_when_literals_uncompressed() { stats.set_literals_compressed_for_tests(false); stats.price_type = HcOptPriceType::Predefined; assert_eq!( - profile.literal_price(&stats, b'a'), + profile.literal_price::(&stats, b'a'), 8 * HC_BITCOST_MULTIPLIER, "uncompressed literals should cost 8 bits regardless of price mode" ); @@ -1337,7 +1340,7 @@ fn dictionary_huffman_seed_ignored_when_literals_uncompressed() { stats.seed_dictionary_entropy(Some(&huff), Some(&*ll), Some(&*ml), Some(&*of)); stats.rescale_freqs( b"abcd", - HcOptimalCostProfile::const_for_strategy::(), + ::ACCURATE_PRICE, ); assert_eq!( stats.lit_sum, 0, @@ -1667,6 +1670,51 @@ fn hc_ldm_candidates_are_merged_into_optimal_candidates() { ); } +/// A repeat match past the sufficient length ends the search early, and the +/// long-distance candidate must still join afterwards: upstream adds it after +/// `ZSTD_btGetAllMatches` whatever the search did (zstd_opt.c, +/// `ZSTD_optLdm_processMatchCandidate`). Merged inside the search, it was lost +/// on every early exit. +#[test] +fn hc_ldm_candidate_survives_the_search_early_exit() { + let mut hc = HcMatchGenerator::new(512); + hc.strategy_tag = crate::encoding::strategy::StrategyTag::BtOpt; + // rep0 = 10 matches `abcde` at position 10, then `Y` meets `X`: length 5. + hc.table.history = b"abcdeXXXXXabcdeYYYYYYYYYYYYYYYYYYYY".to_vec(); + hc.table.history_start = 0; + hc.table.history_abs_start = 0; + hc.table.search_depth = 32; + + let abs_pos = 10usize; + let ldm = MatchCandidate { + start: abs_pos, + offset: 7, + match_len: 12, + }; + let mut out = Vec::new(); + hc.collect_optimal_candidates( + abs_pos, + hc.table.history.len(), + // Below the repeat match's length, so the repeat probe ends the search. + 4, + HcCandidateQuery { + reps: [10, 20, 30], + lit_len: 1, + ldm_candidate: Some(ldm), + }, + &mut out, + ); + assert!( + out.iter().any(|c| c.offset == 10 && c.match_len == 5), + "the repeat match that ends the search is kept" + ); + assert_eq!( + out.last().map(|c| (c.offset, c.match_len)), + Some((ldm.offset, ldm.match_len)), + "the longer long-distance candidate joins after the early exit" + ); +} + #[test] fn btultra_and_btultra2_both_keep_dictionary_candidates() { // Routes the BtUltra2 / BtUltra fixture through the production diff --git a/zstd/src/encoding/match_table/storage.rs b/zstd/src/encoding/match_table/storage.rs index bc979e72f..68de61256 100644 --- a/zstd/src/encoding/match_table/storage.rs +++ b/zstd/src/encoding/match_table/storage.rs @@ -20,7 +20,7 @@ use alloc::vec::Vec; use super::super::Sequence; use super::super::blocks::encode_offset_with_history; -use super::super::cost_model::{HC_OPT_NUM, HcOptimalCostProfile}; +use super::super::cost_model::HC_OPT_NUM; use super::super::dict_attach::DictAttach; use super::super::hc::HC_MIN_MATCH_LEN; use super::super::opt::types::{HcOptimalSequence, MatchCandidate}; @@ -160,6 +160,20 @@ pub(crate) const HC_CHAIN_LOG: usize = 19; /// modes leave it sized to zero. pub(crate) const HC3_HASH_LOG: usize = 17; +/// The binary tree's coordinates: the biases that map a stored index to its +/// absolute position (`abs_bias`), its history index (`idx_bias`) and its pair +/// slot (`bt_bias`, then `& bt_mask`, doubled). They follow from the position +/// base, the index shift, the history start and the chain log, none of which +/// moves while an armed block is parsed, so the parser takes them once per +/// block, as upstream resolves `base` and `btMask` once per call. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) struct BtCoords { + pub(crate) abs_bias: usize, + pub(crate) idx_bias: usize, + pub(crate) bt_bias: usize, + pub(crate) bt_mask: usize, +} + /// Shared storage backing every match finder. Holds the contiguous /// Immutable dictionary match structure (upstream zstd `ZSTD_dictMatchState`) for /// the binary-tree / optimal path. A hash + single-link chain over the @@ -283,6 +297,9 @@ pub(crate) struct MatchTable { /// guarantee 8 readable bytes); the HC `hash_position` stays 4-byte. /// Defaults to `4`. pub(crate) search_mls: usize, + /// Tree coordinates of the block being parsed, taken by + /// [`Self::capture_block_coords`] before the parse. + pub(crate) block_coords: BtCoords, /// Immutable dictionary match chain (upstream zstd `ZSTD_dictMatchState`), /// searched by the BT/optimal collect alongside the live tree. `Some` /// once primed from a non-empty dictionary on a BT level. @@ -340,6 +357,7 @@ impl Clone for MatchTable { is_btultra2: self.is_btultra2, uses_bt: self.uses_bt, search_mls: self.search_mls, + block_coords: self.block_coords, dms: self.dms.clone(), borrowed_input: self.borrowed_input, borrowed_block: self.borrowed_block, @@ -382,6 +400,7 @@ impl Clone for MatchTable { self.is_btultra2 = source.is_btultra2; self.uses_bt = source.uses_bt; self.search_mls = source.search_mls; + self.block_coords = source.block_coords; self.borrowed_input = source.borrowed_input; self.borrowed_block = source.borrowed_block; self.kernel = source.kernel; @@ -512,6 +531,7 @@ impl MatchTable { is_btultra2: false, uses_bt: false, search_mls: 4, + block_coords: BtCoords::default(), dms: DictAttach::new(), borrowed_input: None, borrowed_block: None, @@ -1426,6 +1446,26 @@ impl MatchTable { (1usize << self.bt_log()) - 1 } + /// The binary tree's coordinates as the table stands now. + pub(crate) fn bt_coords(&self) -> BtCoords { + let abs_bias = self + .position_base + .wrapping_sub(1) + .wrapping_sub(self.index_shift); + BtCoords { + abs_bias, + idx_bias: abs_bias.wrapping_sub(self.history_abs_start), + bt_bias: self.position_base.wrapping_sub(1), + bt_mask: self.bt_mask(), + } + } + + /// Take the tree's coordinates for the block about to be parsed; the + /// search reads them from [`Self::block_coords`] on every position. + pub(crate) fn capture_block_coords(&mut self) { + self.block_coords = self.bt_coords(); + } + /// Convert an absolute position into a BT pair index in /// `chain_table`. Each node occupies two consecutive slots /// (smaller, larger) so the result is doubled. Upstream zstd parity: @@ -1679,17 +1719,17 @@ impl MatchTable { (start_cursor, start_cursor) } - /// Stage D: BT walker step. Cross-platform dispatcher that picks - /// the per-kernel variant so the per-iteration - /// `count_match_from_indices` symbol inlines under the kernel's - /// `target_feature` umbrella. Previously lived on `BtMatcher` - /// but the body uses only table state plus `self.search_depth`, - /// so it migrates onto `MatchTable` and clears the cross-struct - /// borrow that blocked the rest of the BT update chain. + /// Insert the positions `[from, stop)` into the binary tree, with the + /// window floor taken at `target_abs`, and return the cursor after the + /// last insertion (a long match can carry it past `stop`). Cross-platform + /// dispatcher: the per-kernel variant runs the whole range under its + /// `target_feature` umbrella. Every position in the range must already be + /// representable (see [`Self::arm_block_positions`]). #[inline(always)] - pub(crate) fn bt_insert_step_no_rebase( + pub(crate) fn bt_insert_range( &mut self, - abs_pos: usize, + from: usize, + stop: usize, current_abs_end: usize, target_abs: usize, ) -> usize { @@ -1699,7 +1739,7 @@ impl MatchTable { feature = "kernel-neon" ))] unsafe { - self.bt_insert_step_no_rebase_neon(abs_pos, current_abs_end, target_abs) + self.bt_insert_range_neon(from, stop, current_abs_end, target_abs) } #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] { @@ -1707,14 +1747,14 @@ impl MatchTable { match self.kernel { #[cfg(feature = "kernel-avx2")] FastpathKernel::Avx2Bmi2 => unsafe { - self.bt_insert_step_no_rebase_avx2_bmi2(abs_pos, current_abs_end, target_abs) + self.bt_insert_range_avx2_bmi2(from, stop, current_abs_end, target_abs) }, #[cfg(feature = "kernel-sse")] FastpathKernel::Sse2 | FastpathKernel::Sse42 => unsafe { - self.bt_insert_step_no_rebase_sse2(abs_pos, current_abs_end, target_abs) + self.bt_insert_range_sse2(from, stop, current_abs_end, target_abs) }, FastpathKernel::Scalar => { - self.bt_insert_step_no_rebase_scalar(abs_pos, current_abs_end, target_abs) + self.bt_insert_range_scalar(from, stop, current_abs_end, target_abs) } } } @@ -1728,7 +1768,7 @@ impl MatchTable { // SAFETY: the `cfg` above establishes `simd128` at compile time, which // is exactly the umbrella the callee declares. unsafe { - self.bt_insert_step_no_rebase_simd128(abs_pos, current_abs_end, target_abs) + self.bt_insert_range_simd128(from, stop, current_abs_end, target_abs) } #[cfg(not(any( all( @@ -1745,11 +1785,11 @@ impl MatchTable { ) )))] { - self.bt_insert_step_no_rebase_scalar(abs_pos, current_abs_end, target_abs) + self.bt_insert_range_scalar(from, stop, current_abs_end, target_abs) } } - /// NEON umbrella BT walker step. + /// NEON umbrella binary-tree range insertion. /// /// # Safety /// AArch64 with NEON (baseline). @@ -1759,24 +1799,26 @@ impl MatchTable { feature = "kernel-neon" ))] #[target_feature(enable = "neon")] - pub(crate) unsafe fn bt_insert_step_no_rebase_neon( + pub(crate) unsafe fn bt_insert_range_neon( &mut self, - abs_pos: usize, + from: usize, + stop: usize, current_abs_end: usize, target_abs: usize, ) -> usize { let search_depth = self.search_depth; - super::super::hc::generator::bt_insert_step_no_rebase_body!( + super::super::hc::generator::bt_insert_range_body!( self, search_depth, - abs_pos, + from, + stop, current_abs_end, target_abs, crate::encoding::fastpath::neon::count_match_from_indices ) } - /// SSE2 umbrella BT walker step. + /// SSE2 umbrella binary-tree range insertion. /// /// # Safety /// x86/x86_64 with SSE2. @@ -1785,24 +1827,26 @@ impl MatchTable { feature = "kernel-sse" ))] #[target_feature(enable = "sse2")] - pub(crate) unsafe fn bt_insert_step_no_rebase_sse2( + pub(crate) unsafe fn bt_insert_range_sse2( &mut self, - abs_pos: usize, + from: usize, + stop: usize, current_abs_end: usize, target_abs: usize, ) -> usize { let search_depth = self.search_depth; - super::super::hc::generator::bt_insert_step_no_rebase_body!( + super::super::hc::generator::bt_insert_range_body!( self, search_depth, - abs_pos, + from, + stop, current_abs_end, target_abs, crate::encoding::fastpath::sse2::count_match_from_indices ) } - /// AVX2+BMI2 umbrella BT walker step. + /// AVX2+BMI2 umbrella binary-tree range insertion. /// /// # Safety /// x86/x86_64 with AVX2 + BMI2. @@ -1811,24 +1855,26 @@ impl MatchTable { feature = "kernel-avx2" ))] #[target_feature(enable = "avx2,bmi2")] - pub(crate) unsafe fn bt_insert_step_no_rebase_avx2_bmi2( + pub(crate) unsafe fn bt_insert_range_avx2_bmi2( &mut self, - abs_pos: usize, + from: usize, + stop: usize, current_abs_end: usize, target_abs: usize, ) -> usize { let search_depth = self.search_depth; - super::super::hc::generator::bt_insert_step_no_rebase_body!( + super::super::hc::generator::bt_insert_range_body!( self, search_depth, - abs_pos, + from, + stop, current_abs_end, target_abs, crate::encoding::fastpath::avx2_bmi2::count_match_from_indices ) } - /// WebAssembly `simd128` umbrella BT walker step. + /// WebAssembly `simd128` umbrella binary-tree range insertion. /// /// # Safety /// wasm32 with `simd128` enabled at compile time. @@ -1838,640 +1884,89 @@ impl MatchTable { feature = "kernel-simd128" ))] #[target_feature(enable = "simd128")] - pub(crate) unsafe fn bt_insert_step_no_rebase_simd128( + pub(crate) unsafe fn bt_insert_range_simd128( &mut self, - abs_pos: usize, + from: usize, + stop: usize, current_abs_end: usize, target_abs: usize, ) -> usize { let search_depth = self.search_depth; - super::super::hc::generator::bt_insert_step_no_rebase_body!( + super::super::hc::generator::bt_insert_range_body!( self, search_depth, - abs_pos, + from, + stop, current_abs_end, target_abs, crate::encoding::fastpath::simd128::count_match_from_indices ) } - /// Scalar fallback BT walker step. Compiled unless the NEON tier covers - /// this target, i.e. on every non-AArch64 target and on AArch64 when - /// `kernel-neon` is off. + /// Scalar fallback binary-tree range insertion. Compiled unless the NEON + /// tier covers this target, i.e. on every non-AArch64 target and on + /// AArch64 when `kernel-neon` is off. #[cfg(not(all( target_arch = "aarch64", target_endian = "little", feature = "kernel-neon" )))] - pub(crate) fn bt_insert_step_no_rebase_scalar( + pub(crate) fn bt_insert_range_scalar( &mut self, - abs_pos: usize, + from: usize, + stop: usize, current_abs_end: usize, target_abs: usize, ) -> usize { let search_depth = self.search_depth; - super::super::hc::generator::bt_insert_step_no_rebase_body!( + super::super::hc::generator::bt_insert_range_body!( self, search_depth, - abs_pos, + from, + stop, current_abs_end, target_abs, crate::encoding::fastpath::scalar::count_match_from_indices ) } - /// Stage D: cross-platform dispatcher for the BT collect-matches walker. - /// External / test entry — the hot path bypasses this and calls the - /// per-kernel variant from inside the surrounding - /// `collect_optimal_candidates_initialized_` umbrella. - #[allow(dead_code)] - #[allow(clippy::too_many_arguments)] - #[inline(always)] - pub(crate) fn bt_insert_and_collect_matches( - &mut self, - abs_pos: usize, - current_abs_end: usize, - profile: &HcOptimalCostProfile, - min_match_len: usize, - best_len_for_skip: &mut usize, - out: &mut Vec, - reps: [u32; 3], - lit_len: usize, - use_hash3: bool, - ) { - #[cfg(all( - target_arch = "aarch64", - target_endian = "little", - feature = "kernel-neon" - ))] - unsafe { - self.bt_insert_and_collect_matches_neon( - abs_pos, - current_abs_end, - profile, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - ) - } - #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] - { - use crate::encoding::fastpath::FastpathKernel; - match self.kernel { - #[cfg(feature = "kernel-avx2")] - FastpathKernel::Avx2Bmi2 => unsafe { - self.bt_insert_and_collect_matches_avx2_bmi2( - abs_pos, - current_abs_end, - profile, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - ) - }, - #[cfg(feature = "kernel-sse")] - FastpathKernel::Sse2 | FastpathKernel::Sse42 => unsafe { - self.bt_insert_and_collect_matches_sse2( - abs_pos, - current_abs_end, - profile, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - ) - }, - FastpathKernel::Scalar => self.bt_insert_and_collect_matches_scalar( - abs_pos, - current_abs_end, - profile, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - ), - } - } - // wasm resolves `simd128` at compile time (no runtime detection), so - // the tier comes from `cfg`, not from `self.kernel`. - #[cfg(all( - target_arch = "wasm32", - target_feature = "simd128", - feature = "kernel-simd128" - ))] - // SAFETY: the `cfg` above establishes `simd128` at compile time, which - // is exactly the umbrella the callee declares. - unsafe { - self.bt_insert_and_collect_matches_simd128( - abs_pos, - current_abs_end, - profile, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - ) - } - #[cfg(not(any( - all( - target_arch = "aarch64", - target_endian = "little", - feature = "kernel-neon" - ), - target_arch = "x86", - target_arch = "x86_64", - all( - target_arch = "wasm32", - target_feature = "simd128", - feature = "kernel-simd128" - ) - )))] - { - self.bt_insert_and_collect_matches_scalar( - abs_pos, - current_abs_end, - profile, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - ) - } - } - - /// NEON-umbrella variant of `bt_insert_and_collect_matches`. - /// - /// # Safety - /// AArch64 with NEON (baseline). - #[cfg(all( - target_arch = "aarch64", - target_endian = "little", - feature = "kernel-neon" - ))] - #[target_feature(enable = "neon")] - #[allow(clippy::too_many_arguments)] - pub(crate) unsafe fn bt_insert_and_collect_matches_neon( - &mut self, - abs_pos: usize, - current_abs_end: usize, - profile: &HcOptimalCostProfile, - min_match_len: usize, - best_len_for_skip: &mut usize, - out: &mut Vec, - reps: [u32; 3], - lit_len: usize, - use_hash3: bool, - ) { - let search_depth = self.search_depth; - super::super::hc::generator::bt_insert_and_collect_matches_body!( - self, - search_depth, - abs_pos, - current_abs_end, - profile.sufficient_match_len, - profile.max_chain_depth, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - crate::encoding::fastpath::neon::common_prefix_len_ptr, - crate::encoding::fastpath::neon::count_match_from_indices, - ) - } - - /// SSE2 umbrella variant of `bt_insert_and_collect_matches`. - /// - /// # Safety - /// x86/x86_64 with SSE2. - #[cfg(all( - any(target_arch = "x86", target_arch = "x86_64"), - feature = "kernel-sse" - ))] - #[target_feature(enable = "sse2")] - #[allow(clippy::too_many_arguments)] - pub(crate) unsafe fn bt_insert_and_collect_matches_sse2( - &mut self, - abs_pos: usize, - current_abs_end: usize, - profile: &HcOptimalCostProfile, - min_match_len: usize, - best_len_for_skip: &mut usize, - out: &mut Vec, - reps: [u32; 3], - lit_len: usize, - use_hash3: bool, - ) { - let search_depth = self.search_depth; - super::super::hc::generator::bt_insert_and_collect_matches_body!( - self, - search_depth, - abs_pos, - current_abs_end, - profile.sufficient_match_len, - profile.max_chain_depth, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - crate::encoding::fastpath::sse2::common_prefix_len_ptr, - crate::encoding::fastpath::sse2::count_match_from_indices, - ) - } - - /// AVX2+BMI2 umbrella variant of `bt_insert_and_collect_matches`. - /// - /// # Safety - /// x86/x86_64 with AVX2 + BMI2. - #[cfg(all( - any(target_arch = "x86", target_arch = "x86_64"), - feature = "kernel-avx2" - ))] - #[target_feature(enable = "avx2,bmi2")] - #[allow(clippy::too_many_arguments)] - pub(crate) unsafe fn bt_insert_and_collect_matches_avx2_bmi2( - &mut self, - abs_pos: usize, - current_abs_end: usize, - profile: &HcOptimalCostProfile, - min_match_len: usize, - best_len_for_skip: &mut usize, - out: &mut Vec, - reps: [u32; 3], - lit_len: usize, - use_hash3: bool, - ) { - let search_depth = self.search_depth; - super::super::hc::generator::bt_insert_and_collect_matches_body!( - self, - search_depth, - abs_pos, - current_abs_end, - profile.sufficient_match_len, - profile.max_chain_depth, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - crate::encoding::fastpath::avx2_bmi2::common_prefix_len_ptr, - crate::encoding::fastpath::avx2_bmi2::count_match_from_indices, - ) - } - - /// WebAssembly `simd128` umbrella BT collect-matches walker. - /// - /// # Safety - /// wasm32 with `simd128` enabled at compile time. - #[cfg(all( - target_arch = "wasm32", - target_feature = "simd128", - feature = "kernel-simd128" - ))] - #[target_feature(enable = "simd128")] - #[allow(clippy::too_many_arguments)] - pub(crate) unsafe fn bt_insert_and_collect_matches_simd128( - &mut self, - abs_pos: usize, - current_abs_end: usize, - profile: &HcOptimalCostProfile, - min_match_len: usize, - best_len_for_skip: &mut usize, - out: &mut Vec, - reps: [u32; 3], - lit_len: usize, - use_hash3: bool, - ) { - let search_depth = self.search_depth; - super::super::hc::generator::bt_insert_and_collect_matches_body!( - self, - search_depth, - abs_pos, - current_abs_end, - profile.sufficient_match_len, - profile.max_chain_depth, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - crate::encoding::fastpath::simd128::common_prefix_len_ptr, - crate::encoding::fastpath::simd128::count_match_from_indices, - ) - } - - /// Scalar fallback BT collect-matches walker. Compiled unless the NEON - /// tier covers this target. - #[cfg(not(all( - target_arch = "aarch64", - target_endian = "little", - feature = "kernel-neon" - )))] - #[allow(clippy::too_many_arguments)] - pub(crate) fn bt_insert_and_collect_matches_scalar( - &mut self, - abs_pos: usize, - current_abs_end: usize, - profile: &HcOptimalCostProfile, - min_match_len: usize, - best_len_for_skip: &mut usize, - out: &mut Vec, - reps: [u32; 3], - lit_len: usize, - use_hash3: bool, - ) { - let search_depth = self.search_depth; - super::super::hc::generator::bt_insert_and_collect_matches_body!( - self, - search_depth, - abs_pos, - current_abs_end, - profile.sufficient_match_len, - profile.max_chain_depth, - min_match_len, - best_len_for_skip, - out, - reps, - lit_len, - use_hash3, - crate::encoding::fastpath::scalar::common_prefix_len_ptr, - crate::encoding::fastpath::scalar::count_match_from_indices, - ) - } - /// BT-side history replay after [`Self::begin_rebase`]. Re-walks - /// `history_start..abs_pos` through the BT step so the pointer-pair + /// `history_start..abs_pos` through the tree insertion so the pointer-pair /// table is consistent with the freshly reset `position_base`. pub(crate) fn replay_history_for_rebase_bt(&mut self, history_start: usize, abs_pos: usize) { let rebuild_end = self.history_abs_end(); - let mut pos = history_start; - while pos < abs_pos { - let forward = self.bt_insert_step_no_rebase(pos, rebuild_end, abs_pos); - // `pos` is a frame-lifetime absolute cursor that can approach - // `usize::MAX` on long 32-bit streams. Cap the step at the - // remaining distance to `abs_pos` so the addition stays - // within `usize` even when the BT walker returns a large - // `forward` near the stream end. - let step = forward.max(1).min(abs_pos - pos); - pos += step; - } - } - - /// Stage D: BT-tree update dispatcher. Picks the kernel-specific - /// variant so the per-iteration BT walker inlines under the - /// surrounding `target_feature` umbrella. - #[inline(always)] - pub(crate) fn bt_update_tree_until(&mut self, abs_pos: usize, current_abs_end: usize) { - #[cfg(all( - target_arch = "aarch64", - target_endian = "little", - feature = "kernel-neon" - ))] - unsafe { - self.bt_update_tree_until_neon(abs_pos, current_abs_end) - } - #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] - { - use crate::encoding::fastpath::FastpathKernel; - match self.kernel { - #[cfg(feature = "kernel-avx2")] - FastpathKernel::Avx2Bmi2 => unsafe { - self.bt_update_tree_until_avx2_bmi2(abs_pos, current_abs_end) - }, - #[cfg(feature = "kernel-sse")] - FastpathKernel::Sse2 | FastpathKernel::Sse42 => unsafe { - self.bt_update_tree_until_sse2(abs_pos, current_abs_end) - }, - FastpathKernel::Scalar => { - self.bt_update_tree_until_scalar(abs_pos, current_abs_end) - } - } - } - #[cfg(all( - target_arch = "wasm32", - target_feature = "simd128", - feature = "kernel-simd128" - ))] - unsafe { - self.bt_update_tree_until_simd128(abs_pos, current_abs_end) - } - #[cfg(not(any( - all( - target_arch = "aarch64", - target_endian = "little", - feature = "kernel-neon" - ), - all( - target_arch = "wasm32", - target_feature = "simd128", - feature = "kernel-simd128" - ), - target_arch = "x86", - target_arch = "x86_64" - )))] - { - self.bt_update_tree_until_scalar(abs_pos, current_abs_end) - } + let _ = self.bt_insert_range(history_start, abs_pos, rebuild_end, abs_pos); } - /// WebAssembly `simd128` umbrella variant: the per-iteration - /// `bt_insert_step_no_rebase_simd128` inlines into the body because both - /// share the `target_feature = "simd128"` umbrella, so the tree walk runs - /// the same tier as the insert step it drives. + /// Insert every position from the insertion frontier up to `abs_pos` + /// into the binary tree (upstream zstd `ZSTD_updateTree`). /// - /// # Safety - /// wasm32 with `simd128` enabled at compile time. - #[cfg(all( - target_arch = "wasm32", - target_feature = "simd128", - feature = "kernel-simd128" - ))] - #[target_feature(enable = "simd128")] - pub(crate) unsafe fn bt_update_tree_until_simd128( - &mut self, - abs_pos: usize, - current_abs_end: usize, - ) { - if self.skip_insert_until_abs < self.history_abs_start { - self.skip_insert_until_abs = self.history_abs_start; - } - let mut update_abs = self.skip_insert_until_abs; - while update_abs < abs_pos { - if !self.can_skip_rebase_check(abs_pos) { - self.maybe_rebase_positions(update_abs); - } - let forward = unsafe { - self.bt_insert_step_no_rebase_simd128(update_abs, current_abs_end, abs_pos) - }; - // Upstream zstd `ZSTD_updateTree`: no clamp to the target, so a long - // match's covered positions stay out of the tree exactly as C leaves - // them. - update_abs += forward.max(1); - } - self.skip_insert_until_abs = abs_pos; - } - - /// NEON-umbrella variant: per-iteration `bt_insert_step_no_rebase_neon` - /// inlines into the body because both share the - /// `target_feature = "neon"` umbrella. - /// - /// # Safety - /// AArch64 with NEON (baseline). - #[cfg(all( - target_arch = "aarch64", - target_endian = "little", - feature = "kernel-neon" - ))] - #[target_feature(enable = "neon")] - pub(crate) unsafe fn bt_update_tree_until_neon( - &mut self, - abs_pos: usize, - current_abs_end: usize, - ) { - if self.skip_insert_until_abs < self.history_abs_start { - self.skip_insert_until_abs = self.history_abs_start; - } - let mut update_abs = self.skip_insert_until_abs; - while update_abs < abs_pos { - if !self.can_skip_rebase_check(abs_pos) { - self.maybe_rebase_positions(update_abs); - } - // SAFETY: same NEON umbrella; direct call inlines the BT-walk body. - let forward = - unsafe { self.bt_insert_step_no_rebase_neon(update_abs, current_abs_end, abs_pos) }; - // Upstream zstd `ZSTD_updateTree`: `idx += ZSTD_insertBt1(...)` with - // NO clamp to the target. The insert step's `forward` skips the - // positions a long match already covers, so letting it overshoot - // `abs_pos` leaves those positions OUT of the tree exactly as C does - // (`nextToUpdate = target` afterwards). Clamping to `abs_pos` inserted - // those covered positions, giving our tree extra candidates C never - // surfaces (e.g. a longer-but-farther match the optimal parser then - // wrongly forces over a cheaper closer one). - update_abs += forward.max(1); - } - self.skip_insert_until_abs = abs_pos; - } - - /// SSE4.2 umbrella variant. - /// - /// # Safety - /// x86/x86_64 with SSE2. - #[cfg(all( - any(target_arch = "x86", target_arch = "x86_64"), - feature = "kernel-sse" - ))] - #[target_feature(enable = "sse2")] - pub(crate) unsafe fn bt_update_tree_until_sse2( - &mut self, - abs_pos: usize, - current_abs_end: usize, - ) { - if self.skip_insert_until_abs < self.history_abs_start { - self.skip_insert_until_abs = self.history_abs_start; - } - let mut update_abs = self.skip_insert_until_abs; - while update_abs < abs_pos { - if !self.can_skip_rebase_check(abs_pos) { - self.maybe_rebase_positions(update_abs); - } - let forward = - unsafe { self.bt_insert_step_no_rebase_sse2(update_abs, current_abs_end, abs_pos) }; - // Upstream zstd `ZSTD_updateTree`: `idx += ZSTD_insertBt1(...)` with - // NO clamp to the target. The insert step's `forward` skips the - // positions a long match already covers, so letting it overshoot - // `abs_pos` leaves those positions OUT of the tree exactly as C does - // (`nextToUpdate = target` afterwards). Clamping to `abs_pos` inserted - // those covered positions, giving our tree extra candidates C never - // surfaces (e.g. a longer-but-farther match the optimal parser then - // wrongly forces over a cheaper closer one). - update_abs += forward.max(1); - } - self.skip_insert_until_abs = abs_pos; - } - - /// AVX2+BMI2 umbrella variant. - /// - /// # Safety - /// x86/x86_64 with AVX2 + BMI2. - #[cfg(all( - any(target_arch = "x86", target_arch = "x86_64"), - feature = "kernel-avx2" - ))] - #[target_feature(enable = "avx2,bmi2")] - pub(crate) unsafe fn bt_update_tree_until_avx2_bmi2( - &mut self, - abs_pos: usize, - current_abs_end: usize, - ) { - if self.skip_insert_until_abs < self.history_abs_start { - self.skip_insert_until_abs = self.history_abs_start; - } - let mut update_abs = self.skip_insert_until_abs; - while update_abs < abs_pos { - if !self.can_skip_rebase_check(abs_pos) { - self.maybe_rebase_positions(update_abs); - } - let forward = unsafe { - self.bt_insert_step_no_rebase_avx2_bmi2(update_abs, current_abs_end, abs_pos) - }; - // Upstream zstd `ZSTD_updateTree`: `idx += ZSTD_insertBt1(...)` with - // NO clamp to the target. The insert step's `forward` skips the - // positions a long match already covers, so letting it overshoot - // `abs_pos` leaves those positions OUT of the tree exactly as C does - // (`nextToUpdate = target` afterwards). Clamping to `abs_pos` inserted - // those covered positions, giving our tree extra candidates C never - // surfaces (e.g. a longer-but-farther match the optimal parser then - // wrongly forces over a cheaper closer one). - update_abs += forward.max(1); - } - self.skip_insert_until_abs = abs_pos; - } - - /// Scalar fallback, compiled unless the NEON tier covers this target. - #[cfg(not(all( - target_arch = "aarch64", - target_endian = "little", - feature = "kernel-neon" - )))] - pub(crate) fn bt_update_tree_until_scalar(&mut self, abs_pos: usize, current_abs_end: usize) { + /// A long match carries the cursor past `abs_pos` with no clamp, as + /// upstream's `idx += ZSTD_insertBt1(...)` does, and the frontier is then + /// set to `abs_pos` (`nextToUpdate = target`): the positions the match + /// covered stay out of the tree. Clamping to `abs_pos` inserted them and + /// gave the tree candidates upstream never surfaces (a longer but farther + /// match the optimal parser then forced over a cheaper closer one). + pub(crate) fn bt_update_tree_until(&mut self, abs_pos: usize, current_abs_end: usize) { if self.skip_insert_until_abs < self.history_abs_start { self.skip_insert_until_abs = self.history_abs_start; } let mut update_abs = self.skip_insert_until_abs; - while update_abs < abs_pos { - if !self.can_skip_rebase_check(abs_pos) { - self.maybe_rebase_positions(update_abs); + if update_abs < abs_pos { + if self.can_skip_rebase_check(abs_pos) { + // Every position up to the target is representable: one run. + let _ = self.bt_insert_range(update_abs, abs_pos, current_abs_end, abs_pos); + } else { + // A rebase may be due inside the run, and it moves the + // coordinates the range hoists, so go one position at a time. + while update_abs < abs_pos { + if !self.can_skip_rebase_check(abs_pos) { + self.maybe_rebase_positions(update_abs); + } + update_abs = + self.bt_insert_range(update_abs, update_abs + 1, current_abs_end, abs_pos); + } } - let forward = - self.bt_insert_step_no_rebase_scalar(update_abs, current_abs_end, abs_pos); - // Upstream zstd `ZSTD_updateTree`: `idx += ZSTD_insertBt1(...)` with - // NO clamp to the target. The insert step's `forward` skips the - // positions a long match already covers, so letting it overshoot - // `abs_pos` leaves those positions OUT of the tree exactly as C does - // (`nextToUpdate = target` afterwards). Clamping to `abs_pos` inserted - // those covered positions, giving our tree extra candidates C never - // surfaces (e.g. a longer-but-farther match the optimal parser then - // wrongly forces over a cheaper closer one). - update_abs += forward.max(1); } self.skip_insert_until_abs = abs_pos; } @@ -2852,7 +2347,7 @@ impl MatchTable { let mut pos = current_abs_start; while pos < current_abs_end { self.maybe_rebase_positions(pos); - let _ = self.bt_insert_step_no_rebase(pos, current_abs_end, current_abs_end); + let _ = self.bt_insert_range(pos, pos + 1, current_abs_end, current_abs_end); self.insert_hash3_only_no_rebase(pos); let next = pos.saturating_add(INCOMPRESSIBLE_SKIP_STEP); if next <= pos { @@ -2871,7 +2366,7 @@ impl MatchTable { continue; } self.maybe_rebase_positions(pos); - let _ = self.bt_insert_step_no_rebase(pos, current_abs_end, current_abs_end); + let _ = self.bt_insert_range(pos, pos + 1, current_abs_end, current_abs_end); self.insert_hash3_only_no_rebase(pos); } diff --git a/zstd/src/encoding/match_table/storage/storage_tests.rs b/zstd/src/encoding/match_table/storage/storage_tests.rs index c795c9dca..b5d3906a9 100644 --- a/zstd/src/encoding/match_table/storage/storage_tests.rs +++ b/zstd/src/encoding/match_table/storage/storage_tests.rs @@ -109,7 +109,7 @@ fn skip_matching_bt_dense_routes_through_bt_update_tree() { fn replay_history_for_rebase_bt_walks_inserted_prefix() { let mut t = new_table(64); // Construct a contiguous mirror long enough for the BT walker - // (`bt_insert_step_no_rebase` reads 8-byte prefixes). + // (`bt_insert_range` reads 8-byte prefixes). t.history = vec![0u8; 64]; for (i, slot) in t.history.iter_mut().enumerate() { *slot = (i % 17) as u8; diff --git a/zstd/src/encoding/mod.rs b/zstd/src/encoding/mod.rs index 374eb2d22..8031c68f6 100644 --- a/zstd/src/encoding/mod.rs +++ b/zstd/src/encoding/mod.rs @@ -107,7 +107,7 @@ pub use levels::config::{ }; pub use match_generator::MatchGeneratorDriver; pub use parameters::{ - Bounds, CParameter, CompressionParameters, CompressionParametersBuilder, + Bounds, CParameter, CompressionParameters, CompressionParametersBuilder, LevelParameters, LiteralCompressionMode, ParameterError, Strategy, }; pub use streaming_encoder::{CompressionContext, StreamingEncoder}; diff --git a/zstd/src/encoding/opt/types.rs b/zstd/src/encoding/opt/types.rs index be074596d..243bf38e7 100644 --- a/zstd/src/encoding/opt/types.rs +++ b/zstd/src/encoding/opt/types.rs @@ -40,6 +40,27 @@ pub(crate) struct HcOptimalNode { pub(crate) reps: [u32; 3], } +impl HcOptimalNode { + /// Record a match ending at `cell`: its offset, its length and an empty + /// literal run. The repeat history is left as it was, possibly + /// uninitialised: the forward pass derives it when it reaches the cell, as + /// upstream's does (`ZSTD_compressBlock_opt_generic` stores `mlen`, `off`, + /// `litlen` and the price, and applies `ZSTD_newRep` at `cur`). + /// + /// # Safety + /// `cell` must be valid for writes. + #[inline(always)] + pub(crate) unsafe fn write_match_end(cell: *mut Self, off: u32, mlen: u32) { + // SAFETY: field places of a writable cell; no reference to the whole + // (possibly uninitialised) node is formed. + unsafe { + core::ptr::addr_of_mut!((*cell).off).write(off); + core::ptr::addr_of_mut!((*cell).mlen).write(mlen); + core::ptr::addr_of_mut!((*cell).litlen).write(0); + } + } +} + impl Default for HcOptimalNode { fn default() -> Self { Self { @@ -61,9 +82,10 @@ pub(crate) struct HcOptimalSequence { pub(crate) lit_len: u32, } -/// Inputs to the per-position candidate collection step. Bundled so the -/// `collect_optimal_candidates_initialized_body!` macro can hand-roll the -/// argument list once. +/// One position's search as the test entry `collect_optimal_candidates` takes +/// it: the repeat history, the pending literal run, and an optional +/// long-distance candidate merged after the search. +#[cfg(test)] #[derive(Copy, Clone)] pub(crate) struct HcCandidateQuery { pub(crate) reps: [u32; 3], @@ -131,4 +153,24 @@ pub(crate) struct HcOptimalPlanBuffers { /// searched and the re-entry reads the answer instead of asking again. /// `None` whenever the buffer's contents do not answer any query. pub(crate) candidates_searched_at: Option<(usize, usize)>, + /// Where the block pass stands between segments. Held here, in memory, + /// rather than in locals of the pass: the pass needs it only at segment + /// boundaries, and as locals it stayed live across the whole segment body, + /// taking registers from the DP's own loops. Upstream keeps the same state + /// behind pointers (`rep`, `ms->opt`) for the same reason. + pub(crate) pass: HcBlockPass, +} + +/// The block pass's position between segments: see +/// [`HcOptimalPlanBuffers::pass`]. +#[derive(Copy, Clone, Default)] +pub(crate) struct HcBlockPass { + /// Block offset of the next segment. + pub(crate) cursor: usize, + /// Literals pending before `cursor`. + pub(crate) litlen: usize, + /// Repeat offsets at `cursor`. + pub(crate) reps: [u32; 3], + /// Block offset the statistics update has consumed literals up to. + pub(crate) literals_cursor: usize, } diff --git a/zstd/src/encoding/parameters.rs b/zstd/src/encoding/parameters.rs index d32dbe3cd..cc4f7616c 100644 --- a/zstd/src/encoding/parameters.rs +++ b/zstd/src/encoding/parameters.rs @@ -242,6 +242,76 @@ impl CParameter { } } +/// The match-finder parameters a numeric level selects for a source of a given +/// size, the drop-in equivalent of C zstd's `ZSTD_getCParams`. +/// +/// The size matters: the reference's level table has a row per source-size +/// tier, and the tier changes the strategy as well as the table widths, so one +/// level can run a different match-finder on a small input than on a large +/// one. This is the selection the encoder itself makes for a frame without a +/// dictionary; knobs set through [`CompressionParameters`] override it. +/// +/// # Examples +/// +/// ``` +/// use structured_zstd::encoding::{LevelParameters, Strategy}; +/// +/// // Level 11 is a lazy2 level on a large or unknown-size source... +/// let large = LevelParameters::for_level(11, None, 0); +/// assert_eq!(large.strategy, Strategy::Lazy2); +/// +/// // ...and the optimal parser on one of 16 KiB or less, whose window is +/// // also cut down to the source. +/// let small = LevelParameters::for_level(11, Some(4096), 0); +/// assert_eq!(small.strategy, Strategy::Btopt); +/// assert_eq!(small.window_log, 12); +/// ``` +#[derive(Copy, Clone, Debug, PartialEq, Eq)] +pub struct LevelParameters { + /// Back-reference window, `log2`. C `windowLog`. + pub window_log: u32, + /// Chain / binary-tree table size, `log2`. C `chainLog`. + pub chain_log: u32, + /// Hash table size, `log2`. C `hashLog`. + pub hash_log: u32, + /// Search attempts per position, `log2`. C `searchLog`. + pub search_log: u32, + /// Match-finder hash width, in bytes. C `minMatch`. + pub min_match: u32, + /// Length that ends a search early; on a negative level, the step size + /// minus one. C `targetLength`. + pub target_length: u32, + /// The match-finder strategy. C `strategy`. + pub strategy: Strategy, +} + +impl LevelParameters { + /// Parameters for `level` compressing a source of `source_size` bytes with + /// a `dictionary_size`-byte dictionary (`0` for none). + /// + /// `level` is on the reference's scale: `0` is the default level, levels + /// above [`CompressionLevel::MAX_LEVEL`] clamp to it, and a negative level + /// is an acceleration factor. `None` is a source of unknown size, which is + /// sized as a large one; `Some(0)` is a source that is really empty. + /// C `ZSTD_getCParams` spells unknown as `0`, so a caller porting from it + /// maps `0` to `None`. + pub fn for_level(level: i32, source_size: Option, dictionary_size: usize) -> Self { + let size = source_size.unwrap_or(crate::encoding::cparams::CONTENTSIZE_UNKNOWN); + let cp = crate::encoding::cparams::get_cparams(level, size, dictionary_size); + Self { + window_log: cp.window_log, + chain_log: cp.chain_log, + hash_log: cp.hash_log, + search_log: cp.search_log, + min_match: cp.min_match, + target_length: cp.target_length, + // Every row of the level table names one of the nine strategies. + strategy: Strategy::from_ordinal(cp.strategy) + .expect("the level table only holds strategies 1..=9"), + } + } +} + /// Error returned by [`CompressionParametersBuilder::build`] when a knob /// is set outside its [`CParameter::bounds`]. #[derive(Copy, Clone, Debug, PartialEq, Eq)] diff --git a/zstd/src/encoding/sequence_capture.rs b/zstd/src/encoding/sequence_capture.rs index 96c90978e..3d570808b 100644 --- a/zstd/src/encoding/sequence_capture.rs +++ b/zstd/src/encoding/sequence_capture.rs @@ -17,7 +17,7 @@ //! and its consumer bench only produce the data, not the labels. //! //! Implementation goes through [`FrameCompressor::new_with_matcher`] + -//! a [`CapturingMatcher`] wrapper rather than driving the matcher in +//! a `CapturingMatcher` wrapper rather than driving the matcher in //! isolation, so the captured stream reflects block-splitter decisions, //! strategy-tag selection and per-level resets exactly as the //! production encoder would emit them. Capturing the matcher in @@ -251,11 +251,11 @@ impl Matcher for CapturingMatcher { /// `compressed.len() >= MAX_BLOCK_SIZE`. The capture would then /// contain phantom triples whose on-wire form has no sequences. To /// prevent silently misaligned output, this function parses the -/// emitted frame's block headers (RFC 8878 §3.1.1.2.2) via -/// [`detect_raw_or_rle_blocks_in_frame`] and panics with a clear -/// diagnostic if any Raw_Block or RLE_Block is present. Callers -/// see a hard failure instead of a misleading capture -/// (PR #149 review #25). +/// emitted frame's block headers (RFC 8878 §3.1.1.2.2) and panics with +/// a clear diagnostic if any Raw_Block or RLE_Block is present, or if +/// the frame holds more blocks than the matcher was asked for (a +/// post-split frame). Callers see a hard failure instead of a +/// misleading capture. pub fn compress_and_collect_sequences(input: &[u8], level: CompressionLevel) -> SequenceCapture { compress_and_collect_sequences_impl(input, level, None, None) } @@ -339,7 +339,8 @@ fn compress_and_collect_sequences_impl( // maps to exactly ONE physical on-wire block, so // `CapturingMatcher::current_block` tracks correctly. // - // * Post-split (`Level(16..=22)` + window >= 1<<17, dispatched + // * Post-split (a btopt / btultra / btultra2 frame with a window of at + // least 1<<17, dispatched // from `levels/fastest.rs::compress_block_encoded` via // `compress_block_with_post_split`): a SINGLE matcher call's // output is split into multiple physical blocks by @@ -347,11 +348,10 @@ fn compress_and_collect_sequences_impl( // → N blocks → `current_block` only increments once, // `block_tail_lengths.len()` is short by `N - 1`. // - // Reject `Level(n >= 16)` only. Covers `Level(16..=22)` and - // clamped `Level(>22)` (match_generator.rs:412-415 lands on - // Level 22 params for n > 22). `Level(11..=15)` is allowed - // because pre-split produces a separate matcher call per - // physical block (PR #149 review #24 + #27 + #30). + // Reject `Level(n >= 16)` up front: every such level runs the optimal + // band on a large source, and levels above 22 clamp to 22. A lower level + // the source size or a dictionary moves onto that band is caught after + // compression, by counting the frame's blocks. let post_split = matches!(level, CompressionLevel::Level(n) if n >= 16); assert!( !post_split, @@ -460,10 +460,22 @@ fn compress_and_collect_sequences_impl( // or RLE block is present so the broken precondition surfaces // immediately instead of being misread as a real divergence // (PR #149 review #25). - let raw_or_rle = detect_raw_or_rle_blocks_in_frame(&output).expect( + let (physical_blocks, raw_or_rle) = detect_raw_or_rle_blocks_in_frame(&output).expect( "sequence_capture: failed to parse emitted frame header — refusing to \ return a possibly-misaligned capture without raw-block detection", ); + // The level guard above is a prediction; this is the fact. The post-split + // pass follows the strategy the frame runs, which the source size or a + // dictionary can raise onto the optimal band below level 16, so the frame + // itself is checked for more blocks than the matcher was asked for. + assert_eq!( + physical_blocks, + block_tail_lengths.len(), + "compress_and_collect_sequences does not support post-split levels: the \ + frame holds {physical_blocks} blocks for {} matcher calls, so the per-call \ + block counter cannot line up with it.", + block_tail_lengths.len(), + ); assert!( raw_or_rle.is_empty(), "compress_and_collect_sequences: emitted frame contains {} raw/RLE block(s) at \ @@ -480,14 +492,14 @@ fn compress_and_collect_sequences_impl( } } -/// Walk the emitted Zstandard frame and return the on-wire indices -/// of any Raw_Block or RLE_Block entries (RFC 8878 §3.1.1.2.2). The +/// Walk the emitted Zstandard frame and return its block count and the +/// on-wire indices of any Raw_Block or RLE_Block entries (RFC 8878 §3.1.1.2.2). The /// capture's matcher hook cannot observe the encoder's late /// raw-fallback decision; this parser gives us a way to fail-fast /// when that decision happens. Returns `Err` on malformed frames so /// the caller can panic with a clearer diagnostic than a silent /// short read. -fn detect_raw_or_rle_blocks_in_frame(frame: &[u8]) -> Result, &'static str> { +fn detect_raw_or_rle_blocks_in_frame(frame: &[u8]) -> Result<(usize, Vec), &'static str> { const ZSTD_MAGIC: [u8; 4] = [0x28, 0xB5, 0x2F, 0xFD]; if frame.len() < 6 || frame[..4] != ZSTD_MAGIC { return Err("frame missing zstd magic"); @@ -575,7 +587,7 @@ fn detect_raw_or_rle_blocks_in_frame(frame: &[u8]) -> Result, &'stati if content_checksum_flag == 1 && cursor.checked_add(4).is_none_or(|end| end > frame.len()) { return Err("truncated content checksum"); } - Ok(raw_or_rle) + Ok((block_idx, raw_or_rle)) } #[cfg(test)] diff --git a/zstd/src/encoding/strategy.rs b/zstd/src/encoding/strategy.rs index 1d9cc3e78..32ef81d3d 100644 --- a/zstd/src/encoding/strategy.rs +++ b/zstd/src/encoding/strategy.rs @@ -17,9 +17,10 @@ //! └─ S::USE_BT == true → start_matching_optimal:: //! ├─ HcOptimalCostProfile::const_for_strategy::() //! ├─ should_run_btultra2_seed_pass:: // const false unless S = BtUltra2 -//! └─ select_kernel() once, then per-segment loop: -//! └─ build_optimal_plan_impl_:: -//! └─ build_optimal_plan_impl_body!(S) +//! └─ select_kernel() once, then one pass over the block: +//! └─ run_optimal_block_:: +//! └─ optimal_block_body!: per segment +//! build_optimal_plan_impl_body!(S) //! ├─ S::OPT_LEVEL == 0 → abort_on_worse_match //! ├─ S::OPT_LEVEL >= 2 → opt_level (refined) //! └─ $collect:: diff --git a/zstd/src/fse/fse_encoder.rs b/zstd/src/fse/fse_encoder.rs index 4d6def096..e8cbbc347 100644 --- a/zstd/src/fse/fse_encoder.rs +++ b/zstd/src/fse/fse_encoder.rs @@ -1,21 +1,18 @@ use crate::bit_io::BitWriter; use alloc::vec::Vec; -pub(crate) struct FSEEncoder<'output, V: AsMut>> { - pub(super) table: FSETable, +/// Encodes a stream with a table the caller holds, so a table built into +/// reusable storage is never moved into the encoder. +pub(crate) struct FSEEncoder<'table, 'output, V: AsMut>> { + pub(super) table: &'table FSETable, writer: &'output mut BitWriter, } -impl>> FSEEncoder<'_, V> { - pub fn new(table: FSETable, writer: &mut BitWriter) -> FSEEncoder<'_, V> { +impl<'table, 'output, V: AsMut>> FSEEncoder<'table, 'output, V> { + pub fn new(table: &'table FSETable, writer: &'output mut BitWriter) -> Self { FSEEncoder { table, writer } } - #[cfg(any(test, feature = "fuzz-exports"))] - pub fn into_table(self) -> FSETable { - self.table - } - /// Encodes the data using the provided table /// Writes /// * Table description @@ -180,6 +177,10 @@ pub struct FSETable { pub(super) state_table_flat: [u16; MAX_FSE_TABLE_SIZE], /// Per-symbol upstream zstd-parity coding transform — see [`SymbolTT`]. pub(super) symbol_tt: [SymbolTT; 256], + /// How many leading entries of `states` / `symbol_tt` may hold a non-default + /// value; everything past it is default. Lets a rebuild clear only what the + /// previous build wrote beyond the new alphabet instead of the whole array. + live_symbols: usize, } impl FSETable { @@ -193,6 +194,7 @@ impl FSETable { table_size: 0, state_table_flat: [0u16; MAX_FSE_TABLE_SIZE], symbol_tt: [SymbolTT::default(); 256], + live_symbols: 0, } } @@ -423,6 +425,7 @@ pub fn build_table_from_data( build_table_from_counts(&counts[..=max_symbol], max_log, avoid_0_numbit) } +#[cfg(any(test, feature = "fuzz-exports", feature = "dict-builder"))] pub(crate) fn build_table_from_symbol_counts( counts: &[usize], max_log: u8, @@ -496,6 +499,7 @@ pub(crate) fn build_seq_ctable_into( build_table_from_probabilities_into(&probs[..=max_symbol], table_log, out); } +#[cfg(any(test, feature = "fuzz-exports", feature = "dict-builder"))] fn build_table_from_counts(counts: &[usize], max_log: u8, avoid_0_numbit: bool) -> FSETable { let mut out = FSETable::blank(); build_table_from_counts_into(counts, max_log, avoid_0_numbit, &mut out); @@ -842,15 +846,21 @@ pub(super) fn build_table_from_probabilities_into(probs: &[i32], acc_log: u8, ou states: symbol_states, state_table_flat, symbol_tt, + live_symbols, } = out; *out_table_size = table_size; // The destination may be a table from an earlier block: reset what the // build below does not overwrite in full. `state_table_flat` needs none of // it, the scatter fills every slot under `table_size` and nothing reads // past it; `symbol_tt` and `states` are written only for `probs.len()` - // symbols, so the tail above that has to be cleared rather than inherited. - symbol_states[probs.len()..].fill_with(SymbolStates::default); - symbol_tt[probs.len()..].fill(SymbolTT::default()); + // symbols, so what an earlier build wrote above that has to be cleared + // rather than inherited. Past `live_symbols` the arrays are already + // default, so the clear stops there. + if *live_symbols > probs.len() { + symbol_states[probs.len()..*live_symbols].fill_with(SymbolStates::default); + symbol_tt[probs.len()..*live_symbols].fill(SymbolTT::default()); + } + *live_symbols = probs.len(); // Upstream zstd `FSE_buildCTable_wksp` (lib/compress/fse_compress.c) — build // `nextStateTable` (== `state_table_flat`) once via cumul + spread + @@ -977,6 +987,10 @@ pub(super) fn build_table_from_probabilities_into(probs: &[i32], acc_log: u8, ou for (symbol, &prob) in probs.iter().enumerate() { symbol_states[symbol].probability = prob; if prob == 0 { + // A slot reused from an earlier build may still hold that build's + // start state and bit width for this symbol; an absent one has none. + symbol_states[symbol].start_state = None; + symbol_states[symbol].max_num_bits = None; // Upstream zstd fills `symbolTT` for prob==0 too, so `FSE_getMaxNbBits` // still works (returns `acc_log + 1` for absent symbols). // We don't expose that path, but mirror the value for parity. diff --git a/zstd/src/fse/mod.rs b/zstd/src/fse/mod.rs index d7bcc1e50..ae98719e4 100644 --- a/zstd/src/fse/mod.rs +++ b/zstd/src/fse/mod.rs @@ -49,14 +49,11 @@ pub fn round_trip(data: &[u8]) { } let mut writer = BitWriter::new(); - let mut encoder = FSEEncoder::new( - fse_encoder::build_table_from_data(data.iter().copied(), 6, false), - &mut writer, - ); + let enc_table = fse_encoder::build_table_from_data(data.iter().copied(), 6, false); + let mut encoder = FSEEncoder::new(&enc_table, &mut writer); let mut dec_table = FSETable::new(255); encoder.encode(data); let acc_log = encoder.acc_log(); - let enc_table = encoder.into_table(); let encoded = writer.dump(); let table_bytes = dec_table.build_decoder(&encoded, acc_log).unwrap(); diff --git a/zstd/src/fse/tests.rs b/zstd/src/fse/tests.rs index 112cd4341..d85a4e278 100644 --- a/zstd/src/fse/tests.rs +++ b/zstd/src/fse/tests.rs @@ -21,6 +21,39 @@ fn an_encoder_table_holds_nothing_on_the_heap() { ); } +#[test] +fn rebuilding_a_used_table_matches_a_fresh_build() { + // Block tables are rebuilt in place. A symbol the earlier build had and the + // new one lacks, inside the new alphabet or past it, must read as absent: + // the dictionary cost seed takes `max_num_bits_for_symbol` of an absent + // symbol as zero, and a stale width there prices it as codable. + let mut wide = [0usize; 64]; + for (symbol, count) in wide.iter_mut().enumerate() { + *count = symbol + 1; + } + let mut narrow = [0usize; 40]; + for (symbol, count) in narrow.iter_mut().enumerate() { + *count = if symbol == 5 { 0 } else { 2 * symbol + 3 }; + } + let mut reused = fse_encoder::FSETable::blank(); + fse_encoder::build_table_from_symbol_counts_into(&wide, 9, false, &mut reused); + fse_encoder::build_table_from_symbol_counts_into(&narrow, 9, false, &mut reused); + let fresh = fse_encoder::build_table_from_symbol_counts(&narrow, 9, false); + for symbol in 0..=255u8 { + assert_eq!( + reused.symbol_probability(symbol), + fresh.symbol_probability(symbol), + "probability of symbol {symbol}" + ); + assert_eq!( + reused.max_num_bits_for_symbol(symbol), + fresh.max_num_bits_for_symbol(symbol), + "max bits of symbol {symbol}" + ); + } + assert_eq!(reused.table_header_bits(), fresh.table_header_bits()); +} + #[test] fn decoder_entry_layout_is_four_bytes_for_huffman_weights() { assert_eq!(core::mem::size_of::(), 4); diff --git a/zstd/src/huff0/huff0_decoder/tests.rs b/zstd/src/huff0/huff0_decoder/tests.rs index 22a19a6c3..c39fbd0bd 100644 --- a/zstd/src/huff0/huff0_decoder/tests.rs +++ b/zstd/src/huff0/huff0_decoder/tests.rs @@ -45,10 +45,8 @@ fn build_decoder_rejects_fse_streams_with_256_explicit_weights() { for &w in &weights { counts[w as usize] += 1; } - let mut encoder = FSEEncoder::new( - build_table_from_symbol_counts(&counts, 6, false), - &mut writer, - ); + let table = build_table_from_symbol_counts(&counts, 6, false); + let mut encoder = FSEEncoder::new(&table, &mut writer); encoder.encode_interleaved(&weights); writer.flush(); } diff --git a/zstd/src/huff0/huff0_encoder.rs b/zstd/src/huff0/huff0_encoder.rs index 6f99a2cd4..9ed0897ca 100644 --- a/zstd/src/huff0/huff0_encoder.rs +++ b/zstd/src/huff0/huff0_encoder.rs @@ -1,3 +1,4 @@ +use alloc::boxed::Box; use alloc::vec::Vec; use core::cmp::Ordering; @@ -380,7 +381,8 @@ impl>> HuffmanEncoder<'_, '_, V> { let weights = self.weights_into(&mut buf); let weights = &weights[..weights.len() - 1]; let mut encoded = Vec::new(); - if Self::encode_weight_description_into(weights, &mut encoded) { + let mut fse_table = fse_encoder::FSETable::blank(); + if Self::encode_weight_description_into(weights, &mut encoded, &mut fse_table) { self.writer.write_bits(encoded.len() as u8, 8); self.writer.append_bytes(&encoded); } else { @@ -396,7 +398,8 @@ impl>> HuffmanEncoder<'_, '_, V> { let len = weights.len(); let weights = &buf[..len - 1]; let mut encoded = Vec::new(); - if Self::encode_weight_description_into(weights, &mut encoded) { + let mut fse_table = fse_encoder::FSETable::blank(); + if Self::encode_weight_description_into(weights, &mut encoded, &mut fse_table) { self.writer.write_bits(encoded.len() as u8, 8); self.writer.append_bytes(&encoded); } else { @@ -413,7 +416,13 @@ impl>> HuffmanEncoder<'_, '_, V> { /// rebuilt many times per frame reuses one allocation; on `false` its /// contents are meaningless and the raw nibble description is written /// instead. - fn encode_weight_description_into(weights: &[u8], encoded: &mut Vec) -> bool { + /// `fse_table` is where the weights' FSE table is built; the caller keeps it + /// between tables, as upstream keeps its `HUF_CompressWeightsWksp`. + fn encode_weight_description_into( + weights: &[u8], + encoded: &mut Vec, + fse_table: &mut fse_encoder::FSETable, + ) -> bool { encoded.clear(); if weights.len() <= 2 { return false; @@ -454,11 +463,9 @@ impl>> HuffmanEncoder<'_, '_, V> { encoded.reserve(want - encoded.len()); } { + fse_encoder::build_table_from_symbol_counts_into(&counts, 6, false, fse_table); let mut writer = BitWriter::from(&mut *encoded); - let mut encoder = FSEEncoder::new( - fse_encoder::build_table_from_symbol_counts(&counts, 6, false), - &mut writer, - ); + let mut encoder = FSEEncoder::new(fse_table, &mut writer); encoder.encode_interleaved(weights); writer.flush(); } @@ -817,15 +824,19 @@ impl HuffmanTable { /// std build path: consults the lazy cache to avoid re-encoding the /// weight stream when both planner and emitter call this for the /// same table. no_std build path: recomputes via the direct encoder - /// every call (cache field absent — preserves `Sync`). - pub(crate) fn try_table_description_size(&mut self) -> Option { + /// every call (cache field absent — preserves `Sync`). `fse_table` is the + /// caller's storage for the weights' FSE table. + pub(crate) fn try_table_description_size( + &mut self, + fse_table: &mut fse_encoder::FSETable, + ) -> Option { #[cfg(feature = "std")] { // Encodes on the first call for these contents and caches it, so // the writer that follows reads rather than repeats the work. This // is also where the caching happens at all: it is the only step in // the emit path holding the table mutably. - self.fill_weight_description_from_codes(); + self.fill_weight_description_from_codes(fse_table); if let Some(fse_description) = self.cached_encoded_weight_description() { return Some(fse_description.len() + 1); } @@ -843,7 +854,11 @@ impl HuffmanTable { let len = weights.len(); let weights = &buf[..len - 1]; let mut encoded = Vec::new(); - if HuffmanEncoder::>::encode_weight_description_into(weights, &mut encoded) { + if HuffmanEncoder::>::encode_weight_description_into( + weights, + &mut encoded, + fse_table, + ) { return Some(encoded.len() + 1); } if weights.len() <= 128 { @@ -855,8 +870,11 @@ impl HuffmanTable { } /// Alias for `try_table_description_size` used by call sites that require explicit writeability. - pub(crate) fn writeable_table_description_size(&mut self) -> Option { - self.try_table_description_size() + pub(crate) fn writeable_table_description_size( + &mut self, + fse_table: &mut fse_encoder::FSETable, + ) -> Option { + self.try_table_description_size(fse_table) } /// Owning form of [`Self::weights_into`], for tests that want the weights @@ -884,21 +902,24 @@ impl HuffmanTable { /// which is the whole point: the previous lazy form populated through /// `&self` and so had to hand the cache a freshly allocated one. #[cfg(feature = "std")] - fn fill_weight_description(&mut self, weights: &[u8]) { + fn fill_weight_description(&mut self, weights: &[u8], fse_table: &mut fse_encoder::FSETable) { if self.cached_encoded_weight_description.state != DescriptionState::NotComputed { return; } let cache = &mut self.cached_encoded_weight_description; - cache.state = - if HuffmanEncoder::>::encode_weight_description_into(weights, &mut cache.buf) { - DescriptionState::Encoded(cache.buf.len()) - } else { - DescriptionState::NotEncodable - }; + cache.state = if HuffmanEncoder::>::encode_weight_description_into( + weights, + &mut cache.buf, + fse_table, + ) { + DescriptionState::Encoded(cache.buf.len()) + } else { + DescriptionState::NotEncodable + }; } #[cfg(feature = "std")] - fn fill_weight_description_from_codes(&mut self) { + fn fill_weight_description_from_codes(&mut self, fse_table: &mut fse_encoder::FSETable) { // Before deriving the weights, not after: they cost a pass over the // whole alphabet, and a warm cache needs none of it. if self.cached_encoded_weight_description.state != DescriptionState::NotComputed { @@ -910,7 +931,7 @@ impl HuffmanTable { let weights = self.weights_into(&mut buf); let len = weights.len(); let weights = &buf[..len - 1]; - self.fill_weight_description(weights); + self.fill_weight_description(weights, fse_table); } /// The cached encoding, or `None` when the FSE form was rejected OR has not @@ -1579,6 +1600,9 @@ pub(crate) struct WeightScratch { /// The last table the caller built and threw away, handed back so the next /// build fills its buffers instead of taking new ones. spare_table: Option, + /// Where each table's weight description builds its FSE table. Taken on + /// first use and kept, so a description costs no table allocation or copy. + weight_fse_table: Option>, } impl WeightScratch { @@ -1595,12 +1619,22 @@ impl WeightScratch { + self.work.capacity() * core::mem::size_of::() + self.weights.capacity() * core::mem::size_of::() + self.spare_table.as_ref().map_or(0, HuffmanTable::heap_size) + + self + .weight_fse_table + .as_ref() + .map_or(0, |_| core::mem::size_of::()) } /// Park a table the caller is done with, for the next build to fill. pub(crate) fn recycle(&mut self, table: HuffmanTable) { self.spare_table = Some(table); } + + /// The storage a weight description builds its FSE table in. + pub(crate) fn weight_fse_table(&mut self) -> &mut fse_encoder::FSETable { + self.weight_fse_table + .get_or_insert_with(|| Box::new(fse_encoder::FSETable::blank())) + } } /// [`build_limited_weights`] filling caller-owned scratch. The weights land in @@ -1902,7 +1936,11 @@ pub(crate) fn huf_weight_description_for_test(data: &[u8]) -> (Vec, Vec) weights.pop(); let mut encoded = Vec::new(); assert!( - HuffmanEncoder::>::encode_weight_description_into(&weights, &mut encoded), + HuffmanEncoder::>::encode_weight_description_into( + &weights, + &mut encoded, + &mut fse_encoder::FSETable::blank(), + ), "expected FSE weights", ); let mut description = Vec::with_capacity(encoded.len() + 1); diff --git a/zstd/src/huff0/huff0_encoder/tests.rs b/zstd/src/huff0/huff0_encoder/tests.rs index 58c837739..3cb9e900d 100644 --- a/zstd/src/huff0/huff0_encoder/tests.rs +++ b/zstd/src/huff0/huff0_encoder/tests.rs @@ -88,7 +88,7 @@ fn the_cached_weight_description_counts_as_retained() { let before = table.heap_size(); let cached_len = table - .writeable_table_description_size() + .writeable_table_description_size(&mut fse_encoder::FSETable::blank()) .expect("the full-alphabet fixture caches an encoded description") - 1; @@ -404,7 +404,7 @@ fn cheap_desc_size_proxy_is_conservative_vs_exact() { // `try_table_description_size` trims internally; mirror that // on the proxy call so both score the same slice. let trimmed = &weights[..weights.len() - 1]; - let exact = table.try_table_description_size(); + let exact = table.try_table_description_size(&mut fse_encoder::FSETable::blank()); let proxy = cheap_desc_size_proxy(trimmed); match (proxy, exact) { (Some(p), Some(e)) => { @@ -523,6 +523,9 @@ fn fse_weight_descriptions_roundtrip() { // alphabet such as 4 symbols → weights [1,1,1] produced a description the // decoder rejected. let mut fails: Vec<(usize, u32, alloc::vec::Vec)> = alloc::vec::Vec::new(); + // One table for every case, as a compressor keeps one: each description + // is built over whatever the previous case left in it. + let mut fse_table = fse_encoder::FSETable::blank(); for card in 2usize..=255 { for skew in 0u32..4 { let mut data: Vec = Vec::new(); @@ -553,7 +556,11 @@ fn fse_weight_descriptions_roundtrip() { // for streams it actually FSE-encodes; None means it chose the raw // description (nothing to round-trip). Every Some MUST decode back. let mut encoded = Vec::new(); - if !HuffmanEncoder::>::encode_weight_description_into(&weights, &mut encoded) { + if !HuffmanEncoder::>::encode_weight_description_into( + &weights, + &mut encoded, + &mut fse_table, + ) { continue; } let mut description = Vec::with_capacity(encoded.len() + 1); @@ -608,7 +615,11 @@ fn large_alphabet_weight_description_uses_fse_when_raw_is_unrepresentable() { let mut encoded = Vec::new(); assert!( - HuffmanEncoder::>::encode_weight_description_into(&weights, &mut encoded), + HuffmanEncoder::>::encode_weight_description_into( + &weights, + &mut encoded, + &mut fse_encoder::FSETable::blank(), + ), "FSE weight description must be available when raw weights cannot be represented", ); let mut description = Vec::with_capacity(encoded.len() + 1); @@ -644,7 +655,7 @@ fn cached_encoded_weight_description_is_reused_for_write_table() { } let mut table = HuffmanTable::build_from_data(&data); let desc_size = table - .writeable_table_description_size() + .writeable_table_description_size(&mut fse_encoder::FSETable::blank()) .expect("table description must be writable"); let cached = table .cached_encoded_weight_description() @@ -674,7 +685,9 @@ fn flat_wide_alphabet_has_no_writeable_description() { let alphabet: Vec = (0u8..=255).collect(); let mut table = HuffmanTable::build_from_data(&alphabet); assert!( - table.writeable_table_description_size().is_none(), + table + .writeable_table_description_size(&mut fse_encoder::FSETable::blank()) + .is_none(), "a table this wide and this flat has no representation to write" ); } @@ -710,7 +723,11 @@ fn a_rejected_description_is_recorded_and_the_raw_form_written() { } // The size query is what encodes, and it is what a writer is preceded by. - assert!(table.writeable_table_description_size().is_some()); + assert!( + table + .writeable_table_description_size(&mut fse_encoder::FSETable::blank()) + .is_some() + ); assert_eq!( table.weight_description_state(), DescriptionState::NotEncodable, diff --git a/zstd/src/huff0/mod.rs b/zstd/src/huff0/mod.rs index f22db343b..3e0db2ee6 100644 --- a/zstd/src/huff0/mod.rs +++ b/zstd/src/huff0/mod.rs @@ -26,7 +26,7 @@ pub fn round_trip(data: &[u8]) { let mut writer = BitWriter::new(); let mut encoder_table = huff0_encoder::HuffmanTable::build_from_data(data); encoder_table - .writeable_table_description_size() + .writeable_table_description_size(&mut crate::fse::fse_encoder::FSETable::blank()) .expect("round_trip must only build Huffman tables with a writeable description"); let mut encoder = huff0_encoder::HuffmanEncoder::new(&encoder_table, &mut writer); diff --git a/zstd/src/lib.rs b/zstd/src/lib.rs index d35b2256a..d93fc0eb6 100644 --- a/zstd/src/lib.rs +++ b/zstd/src/lib.rs @@ -154,7 +154,9 @@ pub mod testing { src: u64, dict: usize, ) -> (u32, u32, u32, u32, u32, u32, u32) { - let cp = crate::encoding::cparams::get_cparams_public(level, src, dict); + // `ZSTD_getCParams` spells an unknown source size as 0. + let cp = + crate::encoding::LevelParameters::for_level(level, (src != 0).then_some(src), dict); ( cp.window_log, cp.chain_log, @@ -162,7 +164,7 @@ pub mod testing { cp.search_log, cp.min_match, cp.target_length, - cp.strategy, + cp.strategy.ordinal(), ) } @@ -250,6 +252,20 @@ pub mod testing { crate::dictionary::dict_roundtrip_fixture() } + /// The content the legacy trainer selects, before any header: facade for + /// the `ffi-bench` test that compares it with the tail of the reference's + /// `ZDICT_trainFromBuffer_legacy` dictionary. `None` when the trainer + /// yields nothing. + #[cfg(feature = "dict-builder")] + pub fn legacy_dict_content( + samples: &[u8], + sample_sizes: &[usize], + dict_size: usize, + selectivity: u32, + ) -> Option> { + crate::dictionary::legacy_dict_content(samples, sample_sizes, dict_size, selectivity) + } + pub use crate::blocks::block::BlockType; /// First block's type (raw / rle / compressed) in a frame. Facade over the diff --git a/zstd/src/tests/parameters_test.rs b/zstd/src/tests/parameters_test.rs index 60ee9b559..576da97e7 100644 --- a/zstd/src/tests/parameters_test.rs +++ b/zstd/src/tests/parameters_test.rs @@ -88,6 +88,41 @@ fn empty_override_is_byte_identical_to_level() { } } +/// Once every match-finder knob is set, the base level has nothing left to +/// decide: the frame is the same whichever level the parameters start from, as +/// it is upstream, where the block splitter follows the effective cParams +/// (`ZSTD_resolveBlockSplitterMode`: `strategy >= btopt && windowLog >= 17`). +/// A dfast base level carrying btultra2 knobs used to skip the post-split pass +/// a btultra2 level runs, because that pass was keyed on the level number. +#[test] +fn a_fully_specified_parameter_set_ignores_the_base_level() { + let data = &include_bytes!("../../decodecorpus_files/z000033")[..512 * 1024]; + let frame_from = |level: i32| { + let params = CompressionParameters::builder(CompressionLevel::Level(level)) + .window_log(20) + .chain_log(21) + .hash_log(21) + .search_log(9) + .min_match(3) + .target_length(999) + .strategy(Strategy::Btultra2) + .build() + .unwrap(); + compress_with_parameters(data, ¶ms) + }; + let native = frame_from(22); + for level in [3, 13] { + let frame = frame_from(level); + assert_eq!( + frame.len(), + native.len(), + "level {level} base with btultra2 knobs diverged from level 22", + ); + assert_eq!(frame, native); + } + assert_eq!(decode(&native), data); +} + /// Custom parameters must produce valid (decodable) frames that /// reproduce the input. #[test]