Repository navigation
Turn pbench into a performance regression test - #186
Merged
danielpclark merged 2 commits intoOct 4, 2026
Merged
Conversation
`FasterPath.ruby_arch_bits` used `1.size`, the width of a C `long`, which is 4 bytes on 64-bit Windows, so the pbench header there read "32-bit ruby" for a 64-bit Ruby. The pointer width is what the Rust side measures too, so the two now agree, and a test checks that they do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CXvfgWzuq28ewqT3AyhX4Z
Each benchmark in the pbench suite gets a `min:` floor, the least improvement over Ruby the method must keep, and `rake pbench` now runs as a minitest test that fails when a method measures below it. Since `rake test` includes pbench, a change that makes a method slower now shows up in CI instead of only in a number nobody compares. The floors are about half of the improvements documented in the README, so that the noise of a shared CI runner doesn't trip them but a method falling back towards Ruby's speed does. The methods that mostly wait on the filesystem (children, directory?, entries) only have to stay faster than Ruby, and the _compat methods and the allocate baseline have no floor. On Windows, where join is currently slower than Ruby's own (#185), regressions are reported as a skip rather than enforced. Closes #61 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CXvfgWzuq28ewqT3AyhX4Z
|
Review changes with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #61.
What
test/pbench/pbench_suite.rb: every benchmark gets amin:floor, the least improvement over Ruby (in percent) the method must keep. The suite now ends in aMinitest::Testthat runs the benchmarks and fails when any method measures below its floor, sorake pbench, and thereforerake testand CI, go red on a performance regression instead of only printing a number.test/pbench/pbench.rb:runreturns the results, and a newregressionsmethod compares them with the floors.lib/faster_path.rb:FasterPath.ruby_arch_bitsmeasures the pointer width instead of1.size(the width of a Clong, 4 bytes on 64-bit Windows), which made the pbench header on Windows CI read "32-bit ruby" for a 64-bit Ruby.test/arch_bits_test.rbchecks the Ruby and Rust sides agree.README.md: a paragraph in "Benchmarks" explaining the floors.The floors
About half of the improvements documented in the README, so the noise of a shared CI runner doesn't trip them but a method falling back towards Ruby's speed does:
absolute?,cleanpath_aggressive,cleanpath_conservative,plus,relative?,relative_path_fromhas_trailing_separator?del_trailing_separator,joinadd_trailing_separator,chop_basename,extnamedirnamebasenamechildren,directory?,entries(filesystem-bound)children_compat,entries_compat, allocate baselineentries_compatmeasured -2.5% onceOn Windows the floors are reported but not enforced (the test skips with the list of regressions), because
joinis currently slower than Ruby's own there, see #185. Once that is fixed the skip can go.Verification
With Ruby 2.7.8 on Linux:
bundle exec rake pbenchpasses (1 test, all methods above their floors, output unchanged apart from the test summary). Temporarily settingdirname's floor to 200% makes the task fail with:TEST_MONKEYPATCHES=true WITH_REGRESSION=true ENCODING=true bundle exec rake minitest lintpasses (132 tests, 72 files inspected, no offenses).🤖 Generated with Claude Code
https://claude.ai/code/session_01CXvfgWzuq28ewqT3AyhX4Z
Generated by Claude Code