fearless simd - #268
Merged
Merged
fearless simd#268
Conversation
…d separator collapsing Adds src/simd.rs with runtime dispatched byte scanning kernels, a CLI_TOOLS_SIMD mode switch, kernel microbenchmarks, simd-bench.sh, and docs/simd.md with the Ryzen 7950X results.
…MD skip-ahead in the scanner PrefixIndex turns the per candidate scan of every file into a binary search, making the first grouping pass over 500 files 15 times faster. The scanner jumps to the next byte that can change its state, 15 to 22 percent faster C, Python, and trailing comment scans. docs/simd.md records both.
Bump version to 10.7.0 and update dependencies. Move the shared simd test helpers into a root-level test_helpers module. Document the simd benchmarks and use descriptive names in the dirmove index and benchmarks. Report the end results in docs/simd.md: move the scanner skip-ahead results under the Ryzen 7950X, where they were measured, replace the Apple Silicon run checklist with general rerun steps, summarize what was kept and what stayed scalar, and align the Markdown tables.
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
A critical mode-dependent SIMD correctness issue and multiple benchmark-script defects remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
Open (3)
What changed in this PR
Adds runtime-dispatched SIMD byte scanning and optimizes dirmove prefix grouping.
Changes:
- Integrates SIMD kernels across semantic line breaking and filename processing.
- Replaces quadratic dirmove scans with a sorted prefix index and cached lowercase parts.
- Adds benchmarks, automation, tests, and performance documentation.
| File | Description |
|---|---|
src/simd.rs |
Implements SIMD kernels, runtime dispatch, and tests. |
src/semantic_line_breaks/tokenizer.rs |
Accelerates token scanning. |
src/semantic_line_breaks/string_syntax.rs |
Defines scanner trigger-byte sets. |
src/semantic_line_breaks/scanner.rs |
Adds scanner shortcuts and skip-ahead. |
src/semantic_line_breaks/markdown.rs |
Updates Markdown processing documentation. |
src/semantic_line_breaks/looks_like_code.rs |
Accelerates code-marker detection. |
src/semantic_line_breaks/formatter.rs |
Reformats documentation comments. |
src/lib.rs |
Adds a separator-processing fast path. |
src/dir_move/utils.rs |
Adds indexed matching and optimized contiguity checks. |
src/dir_move/types.rs |
Caches lowercase filename parts. |
src/dir_move/prefix_index.rs |
Implements sorted prefix lookup. |
src/dir_move/mod.rs |
Exports the prefix-index module. |
src/bin/dir_move/dir_move.rs |
Integrates the shared prefix index. |
simd-bench.sh |
Automates SIMD benchmarks and reporting. |
README.md |
Updates user documentation. |
docs/simd.md |
Documents SIMD design and performance results. |
CLAUDE.md |
Documents architecture and benchmarks. |
Cargo.toml |
Adds dependencies and benchmark registration. |
Cargo.lock |
Updates locked dependencies. |
benches/simd.rs |
Adds SIMD microbenchmarks. |
benches/dir_move.rs |
Adds indexed prefix-matching benchmarks. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| /// Number of UTF-8 characters in the text that are not the given ASCII byte. | ||
| #[must_use] | ||
| pub fn count_chars_excluding(text: &str, excluded: u8) -> usize { | ||
| debug_assert!(excluded.is_ascii(), "the excluded byte must be ASCII"); |
| --save-baseline "mode-$mode" "${CRITERION_ARGS[@]}" "$DUPE_FILTER" | ||
| done | ||
| print_magenta "Kernel microbenchmarks" | ||
| cargo bench --bench simd -- --save-baseline kernels "${CRITERION_ARGS[@]}" |
| print() | ||
| print("| group | input | scalar variant | scalar | simd | change |") | ||
| print("|---|---|---|---:|---:|---:|") | ||
| for group in sorted(root.glob("simd_*")): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fearless_simdbyte-scanning kernels with runtime dispatch and scalar benchmarking mode.simd-bench.sh.Performance