feat(cli): --show-default-cparams, --max, -B# and legacy training

.github/workflows/ci.yml

@@ -348,9 +348,10 @@ jobs:
       - name: FFI parity + cross-validation tests
         # The libzstd-linking parity / cross-validation / corpus tests live in
         # `ffi-bench`; `bench-internals` activates the white-box-facade targets
-        # gated behind it. Corpus fixtures resolve from the manifest dir, so no
+        # gated behind it, and `dict-builder` the trainer parity targets.
+        # Corpus fixtures resolve from the manifest dir, so no
         # working-directory is needed.
-        run: cargo nextest run --profile ci -p ffi-bench --features bench-internals
+        run: cargo nextest run --profile ci -p ffi-bench --features bench-internals,dict-builder
 
   cross-i686:
     needs: lint

README.md

@@ -12,7 +12,7 @@
 
 - **Production-grade decoder** — complete [RFC 8878](https://www.rfc-editor.org/rfc/rfc8878) implementation: dictionary-backed streams, raw / RLE / compressed blocks, the full frame format, optional content checksums, runtime-dispatched SIMD kernels (SSE2 / BMI2 / AVX2 / NEON, opt-in AVX-512).
 - **Full-range encoder** — every C-zstd level (`-131072..=22`) produces valid frames decodable by this crate and by upstream C zstd; named presets, per-knob parameter overrides, long-distance matching, streaming via `std::io::Write`.
-- **Dictionaries end to end** — compress and decompress with the same dictionary format C zstd consumes; reusable parsed handles; pure-Rust COVER / FastCOVER training behind the `dict-builder` feature.
+- **Dictionaries end to end** — compress and decompress with the same dictionary format C zstd consumes; reusable parsed handles; pure-Rust COVER / FastCOVER / legacy training behind the `dict-builder` feature.
 - **Wire-compatible both ways** — frames interoperate with C zstd in either direction; interop is enforced in CI against the reference implementation.
 - **`no_std` ready** — the decoder builds with `--no-default-features` for embedded and sandboxed targets.
 - **WebAssembly / npm** — the same codec as an npm package with automatic SIMD selection; no native addons, no postinstall scripts.
@@ -88,9 +88,15 @@ sound archive.
 `--exclude-compressed` skips inputs whose extension names an
 already-compressed format.
 
-Flags that only steer how the work is done (`-T`, `-B`, `--adapt`, ...) are
-accepted and ignored; their values are still validated, so a typo is an error
-rather than silence.
+Flags that only steer how the work is done (`-T`, `--adapt`, `-B` when
+compressing, ...) are accepted and ignored; their values are still validated,
+so a typo is an error rather than silence. `-b` measures every input as frames
+of its own, cut into `-B#` pieces when a size is given, as upstream's
+benchmark does.
+`--max` sets every compression parameter to its hardest end, as upstream's
+does, with the window stopped at 27 (the widest this build decodes), and
+`--show-default-cparams` prints what the level selects for each input in
+upstream's layout.
 `--target-compressed-block-size` does take effect: it bounds what goes into a
 block, so blocks flush sooner. `--long` means `--long=27`, as upstream
 documents, and is capped there: a larger window would produce frames this
@@ -105,16 +111,20 @@ anything but zstd and `--rsyncable`, which needs the worker threads this build
 does not have. `-M` is treated as the safety promise it is: on the
 runs that decode, a limit covering the 128 MiB window, the decoder's buffers
 and the `-D` dictionary is kept and a tighter one is refused rather than
-ignored. Compressing, listing and training allocate no decoder, so the flag is
-accepted there and describes nothing, as upstream has it.
+ignored. Compressing and listing allocate no decoder, so the flag is accepted
+there and describes nothing, as upstream has it; for `--train-legacy` it caps
+the samples loaded, as below.
 
 `--train` and `--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#]` train
 with FastCOVER, the algorithm upstream also defaults to (a knob set to zero
 keeps its default, as upstream reads it), and a bare `--train-cover` trains
 with the COVER trainer. Its tuning, `--train-cover=...`, is refused rather
 than misread: the reference-side parameters name knobs this trainer does not
-have. `--train-legacy` names an algorithm this build does not have and is
-refused. `-D` takes either a dictionary produced by `--train` or any file at
+have. `--train-legacy[=s=#]` (or `-s#`) runs upstream's original trainer, which
+counts samples: they are loaded as upstream loads them (each file one sample of
+up to 128 KiB, or cut into `-B#` pieces, whole samples up to 2 GiB or `-M` when
+that is smaller), and for the same file list the dictionary carries the same
+content as upstream's. `-D` takes either a dictionary produced by `--train` or any file at
 all, which is then used as raw content the way upstream does; such a
 dictionary has no ID, so the same bytes must be supplied when decoding.
 
@@ -248,6 +258,9 @@ in pure Rust:
 - COVER (`create_raw_dict_from_source`) and FastCOVER (`create_fastcover_raw_dict_from_source`) raw dictionaries
 - `finalize_raw_dict` to produce the full zstd dictionary format
 - `create_fastcover_dict_from_source` for train + finalize in one call
+- `create_legacy_dict_from_slice`: upstream's original suffix-array trainer
+  (`ZDICT_trainFromBuffer_legacy`), whose content matches upstream's byte for
+  byte on the same samples
 
 ## Feature flags
 
@@ -261,7 +274,7 @@ in pure Rust:
 | `kernel-simd128` | ✅ | WebAssembly SIMD kernel (needs `-C target-feature=+simd128`) |
 | `kernel-vbmi2` | ❌ | AVX-512 decode kernel (see note below) |
 | `kernel-scalar` | ✅ | Marker for the always-compiled scalar fallback |
-| `dict-builder` | ❌ | Pure-Rust COVER / FastCOVER dictionary training |
+| `dict-builder` | ❌ | Pure-Rust COVER / FastCOVER / legacy dictionary training |
 | `lsm` | ❌ | [Storage-format extensions](#storage-format-extensions) |
 
 Each flag gates its tier wherever that tier exists. `kernel-sse`,

ffi-bench/Cargo.toml

@@ -133,7 +133,12 @@ required-features = ["bench-internals"]
 [[test]]
 name = "dictionary_ffi"
 path = "tests/dictionary_ffi.rs"
-required-features = ["bench-internals"]
+required-features = ["bench-internals", "dict-builder"]
+
+[[test]]
+name = "legacy_trainer_ffi"
+path = "tests/legacy_trainer_ffi.rs"
+required-features = ["bench-internals", "dict-builder"]
 
 [[test]]
 name = "encode_corpus_ffi"

ffi-bench/tests/legacy_trainer_ffi.rs

@@ -0,0 +1,134 @@
+//! The legacy trainer against the reference's `ZDICT_trainFromBuffer_legacy`:
+//! both walk the same suffix array with the same selection rules, so the
+//! content they choose has to be the same bytes. Only the header in front of it
+//! differs, since the entropy tables are built by each side's own finalizer.
+#![cfg(all(feature = "bench-internals", feature = "dict-builder"))]
+
+use structured_zstd::testing::legacy_dict_content;
+use zstd::zstd_safe::zstd_sys;
+
+/// Every sample back to back, and the length of each.
+type Samples = (Vec<u8>, Vec<usize>);
+
+/// The reference's dictionary content for the same corpus: its dictionary with
+/// the header (magic, id, entropy tables, repeat offsets) cut off.
+fn reference_content(
+    samples: &[u8],
+    sizes: &[usize],
+    dict_size: usize,
+    selectivity: u32,
+) -> Option<Vec<u8>> {
+    let mut dict = vec![0u8; dict_size];
+    let params = zstd_sys::ZDICT_legacy_params_t {
+        selectivityLevel: selectivity,
+        zParams: zstd_sys::ZDICT_params_t {
+            compressionLevel: 0,
+            notificationLevel: 0,
+            dictID: 0,
+        },
+    };
+    // SAFETY: every buffer is valid for the length passed with it, and
+    // `sizes` holds `sizes.len()` entries summing to `samples.len()`.
+    let written = unsafe {
+        zstd_sys::ZDICT_trainFromBuffer_legacy(
+            dict.as_mut_ptr().cast(),
+            dict.len(),
+            samples.as_ptr().cast(),
+            sizes.as_ptr(),
+            sizes.len() as u32,
+            params,
+        )
+    };
+    // SAFETY: plain query on a return code.
+    if written == 0 || unsafe { zstd_sys::ZDICT_isError(written) } != 0 {
+        return None;
+    }
+    dict.truncate(written);
+    // SAFETY: `dict` holds `written` bytes of the dictionary just built.
+    let header = unsafe { zstd_sys::ZDICT_getDictHeaderSize(dict.as_ptr().cast(), dict.len()) };
+    // SAFETY: plain query on a return code.
+    assert_eq!(unsafe { zstd_sys::ZDICT_isError(header) }, 0);
+    Some(dict[header..].to_vec())
+}
+
+/// Log lines of a few shapes, one sample each.
+fn log_lines(count: u32) -> Samples {
+    const SHAPES: [&str; 4] = [
+        "ts={i} level=INFO msg=\"flush memtable\" tenant=demo table=orders region=eu-west\n",
+        "ts={i} level=WARN msg=\"slow compaction\" tenant=demo table=users region=us-east\n",
+        "ts={i} level=INFO msg=\"rotate segment\" tenant=acme table=orders region=eu-west\n",
+        "ts={i} level=ERROR msg=\"write stalled\" tenant=acme table=events region=ap-south\n",
+    ];
+    let mut samples = Vec::new();
+    let mut sizes = Vec::new();
+    for i in 0..count {
+        let line = SHAPES[(i % 4) as usize].replace("{i}", &format!("{:08}", i * 7919));
+        sizes.push(line.len());
+        samples.extend_from_slice(line.as_bytes());
+    }
+    (samples, sizes)
+}
+
+/// The systemd unit files under `dict_tests/files`, one sample each.
+fn unit_files() -> Samples {
+    let dir = concat!(env!("CARGO_MANIFEST_DIR"), "/../zstd/dict_tests/files");
+    let mut names: Vec<_> = std::fs::read_dir(dir)
+        .expect("the fixture directory exists")
+        .map(|entry| entry.expect("readable entry").path())
+        .filter(|path| path.extension().is_none_or(|ext| ext != "zst"))
+        .collect();
+    names.sort();
+    let mut samples = Vec::new();
+    let mut sizes = Vec::new();
+    for name in names {
+        let bytes = std::fs::read(&name).expect("readable fixture");
+        if bytes.is_empty() {
+            continue;
+        }
+        sizes.push(bytes.len());
+        samples.extend_from_slice(&bytes);
+    }
+    (samples, sizes)
+}
+
+/// The decodecorpus file cut into 4 KiB samples, as `--train -B4096` cuts it.
+fn corpus_blocks() -> Samples {
+    let bytes = include_bytes!("../../zstd/decodecorpus_files/z000033").to_vec();
+    let sizes = bytes.chunks(4096).map(<[u8]>::len).collect();
+    (bytes, sizes)
+}
+
+/// Same corpus, same selectivity, same size: the same content, byte for byte.
+#[test]
+fn the_legacy_trainer_selects_the_references_content() {
+    let fixtures: [(&str, Samples); 3] = [
+        ("log lines", log_lines(3000)),
+        ("unit files", unit_files()),
+        ("decodecorpus blocks", corpus_blocks()),
+    ];
+    for (name, (samples, sizes)) in &fixtures {
+        for selectivity in [0u32, 4, 9, 12] {
+            for dict_size in [4096usize, 16 * 1024, 112_640] {
+                let ours = legacy_dict_content(samples, sizes, dict_size, selectivity);
+                let theirs = reference_content(samples, sizes, dict_size, selectivity);
+                let (Some(ours), Some(theirs)) = (ours, theirs) else {
+                    panic!("{name} s={selectivity} size={dict_size}: one side trained nothing");
+                };
+                // The reference places its header in front of the content and
+                // lets it overwrite the front when both do not fit, so its
+                // content is ours or a tail of it.
+                assert!(
+                    theirs.len() <= ours.len() && ours.ends_with(&theirs),
+                    "{name} s={selectivity} size={dict_size}: {} content bytes against the \
+                     reference's {}",
+                    ours.len(),
+                    theirs.len(),
+                );
+                assert!(
+                    theirs.len() * 10 >= ours.len() * 9,
+                    "{name} s={selectivity} size={dict_size}: the header should cost a sliver"
+                );
+            }
+        }
+    }
+}