Fix memory and thread-safety issues for free-threaded (GIL=0) Python

[Qubitium]

Summary

Memory- and thread-safety audit of the C extension and Python layer, focused on free-threaded (GIL=0) CPython. Every fix below has a concrete failure interleaving; the highest-severity items are use-after-free / heap-corruption class.

C extension

Area	Bug	Fix
FindIter (pcre2.c)	Iterator lock existed only on Py_GIL_DISABLED builds, but GIL builds release the GIL around pcre2_match for subjects > 256 KiB — two threads sharing one finditer iterator could run pcre2_match concurrently on the same match_data → torn ovector / heap corruption. The raw PyThread lock could also deadlock a free-threaded stop-the-world pause (attached waiter)	Lock on every build: PyMutex on 3.13+ (parks GC-safely); on older GIL builds the lock is acquired with the GIL released
Match-context cache (cache.c)	match_context_cache_acquire returned the thread-local context while it stayed resident in the slot. Interpreter finalization of a daemon thread, a reentrant acquire from a GC finalizer, or a context-cache toggle flip could free/mutate a context mid-pcre2_match → UAF, double free, or silent offset-limit cross-contamination (matches beyond endpos)	Exclusive ownership: detach on acquire, re-store (or free) on release — same protocol match_data already used
Pattern_substitute	pcre2_substitute executes JIT code but skipped the jit_guard serial lock that every other JIT site takes (PYPCRE_FORCE_JIT_LOCK platforms)	Guard the call when the pattern is JIT-enabled
module_exec error path	A failed re-exec (importlib.reload) ran cache_teardown / pcre_error_teardown / jit_support_teardown / pcre_memory_teardown, freeing locks, TSS keys and exception types while live threads from the previous successful import use them → NULL-deref in raise_pcre_error, UAF on freed locks	Teardowns run only if the very first initialization fails; cache_initialize is idempotent; context-cache env toggle applied only on first init
memory.c	(1) Spinlock waiters cleared a flag they didn't own → two threads could run the allocator initializer concurrently and tear the current_alloc/current_free pair (jemalloc malloc paired with PyMem_Free). (2) Teardown dlclosed jemalloc/tcmalloc and reset current_free while allocations (Match ovectors) were still live → free-with-wrong-allocator / call into unmapped code. (3) pcre_free lazily re-initialized the allocator, possibly selecting a different one than produced the pointer	Proper spin-wait (only the owner clears); allocator selection is process-lifetime irreversible; no re-init in pcre_free
Global pattern cache (pattern_cache.c)	Raw non-reentrant PyThread lock held across PyDict_SetItem/eviction DECREFs → free-threaded stop-the-world deadlock (attached waiter) and self-deadlock when a GC finalizer re-enters pcre.compile; mode flag was published before the lock existed; eviction deleted from the order list before the map, letting a failed map delete strand an unevictable entry	Critical sections on the map dict (STW-safe, reentrancy-safe); PyDict_GetItemRef lookups on 3.13+; publish mode after containers exist; map-first eviction; clear empties containers in place
Match_expand	bytearray template scanned via PyByteArray_AS_STRING with no pin — concurrent ba.clear()/resize frees the buffer mid-memchr → UAF read	Snapshot to bytes under the object's critical section
GC	Match/FindIter hold arbitrary user objects (owner arg) and Pattern can hold str/bytes subclasses, but none had tp_traverse → reference cycles leaked permanently	Py_TPFLAGS_HAVE_GC + traverse/clear for all three types
error.c	raise_pcre_error INCREF'd a borrowed dict entry (free-threaded race window) and could INCREF PcreError == NULL	PyDict_GetItemRef on 3.13+, NULL check
atomic_compat.h	ATOMIC_VAR_INIT was removed in C23 — gcc 15+/clang 19+ fail to build	Fallback definition (also fixed the build in this PR's own testing)

Python layer

• parallel_map nested-call deadlock: a task running on a pcre-worker thread that itself calls parallel_map submitted chunks to the same pool and blocked on their futures; with all workers doing this, everyone waits for chunks only those workers can run. Nested calls now execute inline (thread-local worker flag).
• cache.py epoch re-check: the post-compile check compared _THREAD_LOCAL.epoch — a value only the current thread writes — so it could never detect a concurrent clear_cache(); a stale compiled pattern survived a process-wide clear. Now re-reads the global epoch.
• threads.py: the macOS sysctl subprocess probe ran while holding the process-wide pool lock, stalling every thread touching the pool. Now probed outside the lock with atomic publication.

Testing

• CPython 3.14.7 free-threaded (sys._is_gil_enabled() == False) and 3.14.7 GIL builds, Linux x86-64, linked against system PCRE2 10.46.
• Full pytest suite: 838 passed, 11 skipped, 545 subtests passed (benchmark module excluded).
• Targeted multithreaded stress (8 threads each): shared finditer iterator over a >800 KB subject; match/findall/sub/split churn on one pattern (context/match-data cache reuse); concurrent compile + clear_cache; 1000 owner-reference cycles collected by GC; Match.expand on a bytearray mutated concurrently by another thread. No crashes, no errors, no deadlocks.

🤖 Generated with Claude Code