Skip to content

Take the context lock in wrap_bio() as wrap_socket() does - #219

Merged
sethmlarson merged 3 commits into
sethmlarson:mainfrom
bkrabach:fix/wrap-bio-ctx-lock
Sep 29, 2026
Merged

sethmlarson merged 3 commits into
sethmlarson:mainfrom
bkrabach:fix/wrap-bio-ctx-lock

Conversation

@bkrabach

@bkrabach bkrabach commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

_configure_context() mutates the wrapped ssl.SSLContext — on Linux it calls
ctx.set_default_verify_paths(), which writes the context's X509_STORE.
wrap_socket() holds self._ctx_lock across the __enter__() for that reason
(#192 / #193, released in 0.10.4). wrap_bio() still does a bare
with _configure_context(self._ctx):.

That gap matters because wrap_bio() is the path async stacks take — anyio →
httpcore → httpx never reach wrap_socket() — so the unlocked path is the one
async clients actually use.

What we saw

An application that mounts a session by resolving ~40 agents' model roles
concurrently (one asyncio.gather, one HTTPS call each) dies partway through
with no Python traceback and no result:

double free or corruption (fasttop)
Fatal Python error: Aborted

Exit 134, sometimes SIGSEGV/139 with the same stack. With
PYTHONFAULTHANDLER=1, every abort captured had the same shape — 20–37 threads
inside _configure_context, all of them via wrap_bio, none via
wrap_socket
:

Thread 0x0000e7965cf9f180 (most recent call first):
  File ".../truststore/_openssl.py", line 38 in _configure_context
  File "/usr/lib/python3.12/contextlib.py", line 137 in __enter__
  File ".../truststore/_api.py", line 144 in wrap_bio
  File ".../anyio/_backends/_asyncio.py", line 1100 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner

truststore 0.10.4, Python 3.12.3, OpenSSL 3.0.13, Linux aarch64, in a container.

Verification

A/B/A in one disposable container, this branch built as a wheel and installed
over the stock 0.10.4 — same probe, same concurrency, only truststore changing:

condition concurrent attempts sequential attempts aborts
stock 0.10.4 10 6 2 (plus 1 on the very first, unpaired sequential run)
this branch 20 6 0
reverted to stock 0.10.4 10 — 3

Every abort carried the signature above: 25–31 threads in _configure_context,
100% via wrap_bio, 0 via wrap_socket. An earlier independent run of the same
A/B/A got 7 aborts in 15 stock concurrent attempts (and 1 SIGSEGV in 10 stock
sequential), 0 in 20 patched, 2 in 5 after reverting.

The change

Deliberately the same shape wrap_socket() uses, including holding the lock only
across __enter__() and not for the duration of the wrap — so concurrent wraps
and handshakes are not serialized.

Tests

  • test_wrap_bio_configures_context_under_lock — deterministic. Patches
    _configure_context to record whether _ctx_lock is held while the context is
    being configured, and asserts the lock is released again afterwards. It fails
    on main (records False) and passes with this change.
  • test_threading_wrap_bio — a threaded hammer over wrap_bio(), in the style
    of the existing test_threading. Worth being honest that this one is only
    probabilistic: without the lock the failure is a native abort that takes the
    whole test process down rather than a failed assert, so the deterministic test
    above is the real regression guard.

Full suite passes locally on Linux (nox -s test, Python 3.12, with mkcert and
internet access): 61 passed, 2 skipped.

Notes

  • Cache automatic OpenSSL verify path setup #213 (caching the verify-path setup) would reduce how often this mutation
    happens, but the lock looks worth having either way.
  • Race condition on context setting #209 is a different race in the same area — the window after the lock is
    released, where check_hostname/verify_mode are still relaxed. This PR does
    not address that one.
  • Happy to add a CHANGELOG entry, drop the stress test, or reshape any of this —
    whatever fits best.

CI note

The three pypy3.10 jobs fail on this branch exactly as they do on main (its last three runs, Sep 3–5): cryptography cannot build its wheel there because PyO3 0.29 requires PyPy ≥ 3.11 — the failure is in pip install -r dev-requirements.txt, before any test runs. Every CPython job (3.10–3.14 on all three OSes), Lint, and the pip integration job pass, including the new regression test.

_configure_context() mutates the wrapped ssl.SSLContext, so wrap_socket()
holds self._ctx_lock across its __enter__(). wrap_bio() did not, which
means async stacks that use wrap_bio() (anyio -> httpcore -> httpx) can
mutate one shared context from many threads at once.

Adds a deterministic regression test that wrap_bio() configures the
context under the lock, plus a threaded stress test over wrap_bio().

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@bkrabach
bkrabach marked this pull request as ready for review September 6, 2026 04:16

@sethmlarson sethmlarson left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this contribution and the explanation. Could you add an entry to the CHANGELOG?

Comment thread tests/test_threading.py Outdated
@sethmlarson
sethmlarson merged commit 8d19b4d into sethmlarson:main Sep 29, 2026
19 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants