-mm=gc shared library: its async pool is not hooked when only registered threads call it #522

Since #521, a -mm=gc shared library called only on threads the collector already knows never runs GC_enable_threads, so the library's own async thread pool runs coroutines on unregistered threads, with the collector's allocator unlocked.

Background

The async runtime's thread hooks (typescript::asyncgc::threadHooks(), include/TypeScript/AsyncGCThreads.h) are an inline static, so every module has its own copy. A gc shared library links its own copy of the scheduler (static TypeScriptAsyncRuntime.lib), and the only thing that sets that copy's hooks is the library's own GC_enable_threads.

• Before A gc shared library registers the threads its host calls it on #516, a --emit=dll gc build never called GC_enable_threads (GCPass injects it beside GC_init only into __mlir_gctors or main).
• A gc shared library registers the threads its host calls it on #516 made the first __tslang_gc_enter call it, which also hooked the library's pool.
• A gc shared library loads without starting the collector's markers under the loader lock #521 had to stop that for registered threads: on the loading thread it ran inside DllMain and deadlocked (GC_allow_register_threads starts the marker threads and waits for them under the loader lock).

So now the hooks are set only if some unregistered thread calls into the library.

Effect

A gc library with async code, loaded and called from one registered thread (the usual case: the thread GC_init registered at load):

• The pool threads that resume its coroutines are not registered. A collection neither stops nor scans them, so a frame held only in a worker's registers can be freed underneath it.
• GC_need_to_lock stays FALSE, so a worker and the calling thread can walk the collector's free lists with no lock. That is the crash AsyncGCThreads.h describes (about 1 run in 4 at 200k awaited frames).

The default library is one such library if any of its exported paths await on its own scheduler.

Not covered by tests

test-compile-foreign-thread-gc uses only synchronous exports. A reproducer would be a gc library exporting an async function with a long chain of awaits that allocate, called from a C++ host on its loading thread.

Direction

Enable threads from a registered thread once the library has finished loading, not on the first call. Options:

• a flag the library's load sets when it ends (a last-priority global constructor), which __tslang_gc_enter checks on later calls;
• or have the scheduler ask for threads to be enabled (through a hook set without starting anything) before it first hands work to the pool. The dispatching thread is registered and, outside the load, not under the loader lock.

Related: #516, #521, and the marker-start race in a foreign host (filed beside this).