gh-140746: Fix Thread.start() hanging when the thread dies during bootstrap

[Jihaysse]

Problem

Thread.start() waits for the new thread with a bare self._started.wait(). If the new thread dies before _bootstrap_inner() reaches self._started.set(); typically a MemoryError raised while the C-level thread_run() calls _bootstrap under memory pressure (the interpreter prints Exception ignored in thread started by ...), or a SystemExit which thread_run() clears silently → the event is never set and the parent blocks forever. The dead thread also lingers in threading._limbo, so it stays visible in threading.enumerate() while is_alive() is False.

Any threaded server whose dispatcher thread spawns per-request threads (e.g. socketserver.ThreadingMixIn) freezes permanently the first time this fires. This was detected in production on Python 3.12: workers stopped serving after a burst of MemoryErrors, with kill -3 stack dumps showing the dispatcher thread blocked in Thread.start() → self._started.wait(), the request thread perpetually in its initial state, and no corresponding OS thread. Full investigation in gh-140746 (also reported by Patroni).

Reproducer

Deterministic: simulates the MemoryError that thread_run() swallows while calling _bootstrap (the same C code path, PyObject_Call() returning NULL):

import threading

class BrokenBootstrapThread(threading.Thread):
    # Simulates a MemoryError raised before _started.set(): thread_run()
    # catches it ("Exception ignored in thread started by ...") and the
    # thread exits without ever signalling startup.
    def _bootstrap(self):
        raise MemoryError

t = BrokenBootstrapThread(target=lambda: None)
t.start()   # unpatched: hangs here forever
print("started fine:", t.is_alive())

Unpatched (reproduced on 3.12.11 and main): prints the "Exception ignored" traceback, then start() blocks forever. With this PR: start() raises RuntimeError: thread failed to start and the thread is removed from _limbo. The organic reproducer from the issue (progressively lowering RLIMIT_DATA until a real MemoryError lands in the thread's bootstrap) also hangs unpatched and fails cleanly with this PR after ~1300 progressively-starved thread starts.

Fix

Add a C-level thread_started PyEvent to ThreadHandle, mirroring the existing thread_is_exiting:

• _bootstrap_inner() notifies it (via handle._set_started()) right after self._started.set(), so a waiter woken by the child always observes _started as set;
• thread_run() notifies it unconditionally on its exit path — covering a failed PyObject_Call of _bootstrap, the silent SystemExit case, and the _PyThreadState_MustExit() early exit during finalization. _PyEvent_Notify() is allocation-free C and a no-op on an already-set event, so this is safe on every exit path.

Thread.start() waits on the C event (_wait_for_started(): GIL released, interruptible via Py_MakePendingCalls()) and then checks _started: set means the thread started normally (identical behavior to today); unset means the thread died before starting and never will run, so start() removes it from _limbo and raises RuntimeError("thread failed to start"). No timeout, no polling, no new handle states.

Validation

Environment: Ubuntu 24.04 x86-64 (Linux 6.17), gcc 13.3.0; main built both --with-pydebug and --disable-gil; stock 3.12.11 as the unpatched baseline.

• Both reproducers above: hang unpatched, clean RuntimeError + _limbo cleanup with this PR (both builds).
• End-to-end: a ThreadingTCPServer whose request thread dies in bootstrap freezes permanently unpatched; with this PR, handle_error() logs the RuntimeError and the server keeps serving.
• Test suites (both builds): test_threading, test_thread, test_threadedtempfile, test_thread_local_bytecode, test_socketserver -u all, test_concurrent_futures.test_thread_pool, test_queue — all pass.
• Three regression tests added: MemoryError in _bootstrap, MemoryError in _bootstrap_inner, and SystemExit in _bootstrap.

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #33603875 | 📁 Comparing 07d6226 against main (93beea7)

  🔍 Preview build  

2 files changed

± library/threading.html
± whatsnew/changelog.html