Skip to content

Compile member-selectors once instead of per resolution - #2

Closed
ndreno wants to merge 1 commit into
fix/fail-closed-selectors-and-skipafterfrom
perf/precompile-selectors
Closed

ndreno wants to merge 1 commit into
fix/fail-closed-selectors-and-skipafterfrom
perf/precompile-selectors

Conversation

@ndreno

@ndreno ndreno commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Stacked on #1 (base is that branch; the diff here is only the perf change). Merge #1 first, then this rebases onto main.

The lever

A regex member-selector (REQUEST_HEADERS:/^X-/, !REQUEST_COOKIES:/__utm/) was kept as a string and recompiled with the regex crate on every value it was tested against, on every request. CRS carries 162 rules with a !REQUEST_COOKIES:/__utm/ or /_pk_ref/ exclusion, and the exclusion closure recompiles the selector once per resolved cookie, per rule, per request.

Measured in isolation, that recompilation alone:

cookies= 1   selector evals=  162   recompile-each= 1.359 ms   cached= 0.004 ms   saved 99.7%
cookies= 5   selector evals=  810   recompile-each= 3.251 ms   cached= 0.014 ms   saved 99.6%
cookies=20   selector evals= 3240   recompile-each=12.519 ms   cached= 0.061 ms   saved 99.5%

Against the ~2 ms the gateway measures for a whole inspection, this was the dominant cost, hiding in the exclusion path. It also corrects an earlier conclusion in this project's notes ("borrowing bought only 7%; the ~2 ms is genuine rule evaluation") — the recompile, not evaluation, was the bulk of it on requests with cookies.

The change

Compilation now turns each parsed Target into a CompiledTarget whose regex selector is a compiled regex::Regex, built once. Resolution (Variables::resolve, push_map, the exclusion retain, count_of) matches against the compiled form directly and rebuilds nothing. grep confirms no regex::Regex::new remains in the resolve path (was 2 call sites).

The parsed Target/Selector in the AST are untouched, so the sealed rule-set format is unchanged — only the in-memory compiled program gains the new CompiledTarget/CompiledSelector types, which is where the "compile the AST into an evaluable program" design says a compiled automaton belongs.

This also subsumes the selector validation from #1: compiling the selector is what validates it, so check_targets became compile_targets and an uncompilable selector still fails the build with the same InvalidSelector error. Both #1 tests still pass through the new path.

Verification

  • 169 unit tests pass; clippy and fmt clean.
  • Real CRS 4.9.0 still compiles to 590 rules, same 4 detectSQLi/detectXSS refusals, zero unexpected errors.
  • Exclusion semantics preserved through the compiled path: a request carrying __utmz=attack is allowed by a rule that session=attack trips.
  • Both fail-closed behaviors from Refuse invalid selectors and backward skipAfter at compile time #1 survive: invalid selector refused, backward skipAfter refused.
  • The FTW regression job is the end-to-end detection guard.

Not included

The evaluate_step per-value to_vec() copy (a second, smaller perf lever) and the loose whole-collection exclusion prefix are still open follow-ups from the review.

A regex member-selector was stored as a string and recompiled with the regex
crate on every value it was tested against, on every request. The Core Rule Set
carries 162 rules with a `!REQUEST_COOKIES:/__utm/` or `/_pk_ref/` exclusion,
and each recompiles its selector once per resolved cookie, per rule, per
request. Isolated, that recompilation measured at 1.4 ms for a single-cookie
request and 3.3 ms for five cookies, against a whole-inspection budget of about
2 ms: the dominant cost, hiding in the exclusion path.

Compilation now turns each parsed `Target` into a `CompiledTarget` whose regex
selector is a compiled automaton, built once. Resolution matches against it
directly and rebuilds nothing. Caching the same compiled forms drops the
isolated cost by about 99%.

The parsed `Target`/`Selector` are unchanged, so the sealed rule-set format is
untouched; only the in-memory compiled program gains the new types. This also
subsumes the selector validation added for the fail-closed fix: compiling the
selector is what validates it, so an uncompilable selector still fails the
build with the same `InvalidSelector` error.

The Core Rule Set still compiles to 590 rules with the same 4
detectSQLi/detectXSS refusals, and cookie exclusions still exclude: a request
carrying `__utmz` is allowed by a rule that a `session` cookie trips.
@ndreno
ndreno deleted the branch fix/fail-closed-selectors-and-skipafter September 11, 2026 07:43
@ndreno ndreno closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant