Repository navigation
Conversation
A regex member-selector was stored as a string and recompiled with the regex crate on every value it was tested against, on every request. The Core Rule Set carries 162 rules with a `!REQUEST_COOKIES:/__utm/` or `/_pk_ref/` exclusion, and each recompiles its selector once per resolved cookie, per rule, per request. Isolated, that recompilation measured at 1.4 ms for a single-cookie request and 3.3 ms for five cookies, against a whole-inspection budget of about 2 ms: the dominant cost, hiding in the exclusion path. Compilation now turns each parsed `Target` into a `CompiledTarget` whose regex selector is a compiled automaton, built once. Resolution matches against it directly and rebuilds nothing. Caching the same compiled forms drops the isolated cost by about 99%. The parsed `Target`/`Selector` are unchanged, so the sealed rule-set format is untouched; only the in-memory compiled program gains the new types. This also subsumes the selector validation added for the fail-closed fix: compiling the selector is what validates it, so an uncompilable selector still fails the build with the same `InvalidSelector` error. The Core Rule Set still compiles to 590 rules with the same 4 detectSQLi/detectXSS refusals, and cookie exclusions still exclude: a request carrying `__utmz` is allowed by a rule that a `session` cookie trips.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1 (base is that branch; the diff here is only the perf change). Merge #1 first, then this rebases onto main.
The lever
A regex member-selector (
REQUEST_HEADERS:/^X-/,!REQUEST_COOKIES:/__utm/) was kept as a string and recompiled with theregexcrate on every value it was tested against, on every request. CRS carries 162 rules with a!REQUEST_COOKIES:/__utm/or/_pk_ref/exclusion, and the exclusion closure recompiles the selector once per resolved cookie, per rule, per request.Measured in isolation, that recompilation alone:
Against the ~2 ms the gateway measures for a whole inspection, this was the dominant cost, hiding in the exclusion path. It also corrects an earlier conclusion in this project's notes ("borrowing bought only 7%; the ~2 ms is genuine rule evaluation") — the recompile, not evaluation, was the bulk of it on requests with cookies.
The change
Compilation now turns each parsed
Targetinto aCompiledTargetwhose regex selector is a compiledregex::Regex, built once. Resolution (Variables::resolve,push_map, the exclusionretain,count_of) matches against the compiled form directly and rebuilds nothing.grepconfirms noregex::Regex::newremains in the resolve path (was 2 call sites).The parsed
Target/Selectorin the AST are untouched, so the sealed rule-set format is unchanged — only the in-memory compiled program gains the newCompiledTarget/CompiledSelectortypes, which is where the "compile the AST into an evaluable program" design says a compiled automaton belongs.This also subsumes the selector validation from #1: compiling the selector is what validates it, so
check_targetsbecamecompile_targetsand an uncompilable selector still fails the build with the sameInvalidSelectorerror. Both #1 tests still pass through the new path.Verification
__utmz=attackis allowed by a rule thatsession=attacktrips.skipAfterrefused.Not included
The
evaluate_stepper-valueto_vec()copy (a second, smaller perf lever) and the loose whole-collection exclusion prefix are still open follow-ups from the review.