Harden expression sanitization with AST validation

[nevercodecorrect]

Replace regex-based expression filtering with an AST allowlist and evaluate sanitized expressions without Python builtins.

The previous regex-based solution can be bypassed to execute additional code.

Resolve sanitize=None consistently across all compilation and evaluation paths, including disable_cache and direct NumExpr usage.

Preserve supported syntax and compatibility for conditional expressions, identity and membership comparisons, and variable names containing double underscores. Retain native SyntaxError behavior for parser failures and TypeError for unknown functions.

Add regression tests and release notes for the sanitizer changes.

[Copilot]

Pull request overview

This PR hardens NumExpr’s expression sanitization by replacing the prior regex-based filtering with an AST-based allowlist, and by evaluating sanitized expressions with Python builtins disabled. It also standardizes how sanitize=None resolves across cached, disable_cache=True, and direct NumExpr/compiler entry points, with regression tests and release-note updates.

Changes:

• Replace regex sanitizer with AST validation and block non-allowlisted syntax, dunder names, non-.real/.imag attribute access, and calls to unknown functions.
• Disable Python builtins during eval() when sanitization is enabled, and ensure sanitize=None consistently resolves to the secure default (unless NUMEXPR_SANITIZE=0).
• Add regression tests covering bypass attempts, caching interactions, and compatibility behaviors (conditional expressions, identity/membership comparisons, double-underscore variable names).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
RELEASE_NOTES.rst	Documents the sanitizer hardening and resulting error/behavior changes.
numexpr/tests/test_numexpr.py	Adds regression tests for sanitizer bypasses, cache/no-cache consistency, and accepted syntax forms.
numexpr/necompiler.py	Implements AST allowlist sanitizer, builtins-disabled evaluation, and consistent sanitize=None resolution across compilation/evaluation paths.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[FrancescAlted]

Besides copilot review, I'd like to make sure that the new AST parser is not slowing down things wrt the existent regular expression checks.

[nevercodecorrect]

In terms of performance, the AST parser would be slower than the regex-based option. But it provides better sanitization and avoids payload bypassing.
To measure performance, I used numexpr/bench/varying-expr.py, as it directly exercises the modified code region. Pls correct me if I am wrong.
The data shows it is 1.66% slower in the non-cacheable setup and the same in the cacheable setup.

Setup	Numexpr with non-cacheable	Numexpr with cacheable
regex-based	0.726 seconds	0.008 seconds
AST-based	0.738 seconds	0.008 seconds

The small uncached overhead appears to be the tradeoff for replacing the regex blacklist with structural AST validation. The new sanitizer uses an explicit allowlist of expression-language AST nodes, restricts calls to NumExpr’s function table, and permits only .real and .imag attribute access. The AST-based implementation provides a stronger security boundary than enumerating dangerous textual patterns with regular expressions.

Execution command is PYTHONPATH="$PWD" NUMEXPR_MAX_THREADS=1 python ./bench/varying-expr.py

AST-based logs

Number of iterations 100.  Length of the array: 10 
* Numexpr with non-cacheable expressions:  .......... done in 0.74 seconds
* Numexpr with cacheable expressions:  .......... done in 0.008 seconds
* Numpy with non-cacheable expressions:  .......... done in 0.015 seconds
* Numpy with cacheable expressions:  .......... done in 0.014 seconds

Number of iterations 100.  Length of the array: 10 
* Numexpr with non-cacheable expressions:  .......... done in 0.755 seconds
* Numexpr with cacheable expressions:  .......... done in 0.008 seconds
* Numpy with non-cacheable expressions:  .......... done in 0.018 seconds
* Numpy with cacheable expressions:  .......... done in 0.015 seconds

Number of iterations 100.  Length of the array: 10 
* Numexpr with non-cacheable expressions:  .......... done in 0.751 seconds
* Numexpr with cacheable expressions:  .......... done in 0.008 seconds
* Numpy with non-cacheable expressions:  .......... done in 0.014 seconds
* Numpy with cacheable expressions:  .......... done in 0.014 seconds

Number of iterations 100.  Length of the array: 10 
* Numexpr with non-cacheable expressions:  .......... done in 0.72 seconds
* Numexpr with cacheable expressions:  .......... done in 0.008 seconds
* Numpy with non-cacheable expressions:  .......... done in 0.015 seconds
* Numpy with cacheable expressions:  .......... done in 0.015 seconds

Number of iterations 100.  Length of the array: 10 
* Numexpr with non-cacheable expressions:  .......... done in 0.724 seconds
* Numexpr with cacheable expressions:  .......... done in 0.008 seconds
* Numpy with non-cacheable expressions:  .......... done in 0.015 seconds
* Numpy with cacheable expressions:  .......... done in 0.014 seconds

regex-based logs:

Number of iterations 100.  Length of the array: 10 
* Numexpr with non-cacheable expressions:  .......... done in 0.722 seconds
* Numexpr with cacheable expressions:  .......... done in 0.009 seconds
* Numpy with non-cacheable expressions:  .......... done in 0.015 seconds
* Numpy with cacheable expressions:  .......... done in 0.015 seconds

* Numexpr with non-cacheable expressions:  .......... done in 0.726 seconds
* Numexpr with cacheable expressions:  .......... done in 0.008 seconds
* Numpy with non-cacheable expressions:  .......... done in 0.015 seconds
* Numpy with cacheable expressions:  .......... done in 0.014 seconds

* Numexpr with non-cacheable expressions:  .......... done in 0.735 seconds
* Numexpr with cacheable expressions:  .......... done in 0.008 seconds
* Numpy with non-cacheable expressions:  .......... done in 0.014 seconds
* Numpy with cacheable expressions:  .......... done in 0.015 seconds

* Numexpr with non-cacheable expressions:  .......... done in 0.718 seconds
* Numexpr with cacheable expressions:  .......... done in 0.008 seconds
* Numpy with non-cacheable expressions:  .......... done in 0.014 seconds
* Numpy with cacheable expressions:  .......... done in 0.014 seconds

Number of iterations 100.  Length of the array: 10 
* Numexpr with non-cacheable expressions:  .......... done in 0.728 seconds
* Numexpr with cacheable expressions:  .......... done in 0.008 seconds
* Numpy with non-cacheable expressions:  .......... done in 0.014 seconds
* Numpy with cacheable expressions:  .......... done in 0.014 seconds