stdlib: split the promise bridge from the tokio runtime (turnloop P8 lane L)

[proggeramlug]

Part of the turnloop P8 tokio removal, lane L (scripts/tokio_inventory.json group L: perry-stdlib -> tokio).

What this does

perry-stdlib's async-runtime feature was two things welded together: the promise bridge (the settle queue and main-thread pump that every native binding, stdlib or perry-ext-*, resolves its promises through) and tokio's current-thread runtime. The auto-optimize driver force-enabled async-runtime for every stdlib-linking program, so every such binary carried tokio, including programs that never hand tokio a future.

This PR splits them:

	feature	tokio
common/async_bridge.rs: pinning, InflightGuard, both settle queues, GC scanner, ensure_pump_registered, js_stdlib_process_pending, js_stdlib_has_active_handles, pool_for_promise_deferred, new reject_promise_later	async-bridge (new)	no
common/tokio_bridge.rs (new file, moved verbatim): RUNTIME, wait-driver tick, spawn, spawn_for_promise*, block_on, run_one_tick, drive_pending	async-runtime = async-bridge + dep:tokio	yes

tokio_bridge's public names are re-exported through async_bridge, so no caller's path changed. ws.rs, tls.rs, net/, fetch/ are untouched (sibling lanes G #11101 and H #11102 own them).

perry_ffi_async.rs:

• perry_ffi_spawn_async and perry_ffi_spawn_blocking_with_reactor compile only under async-runtime, because their payload is a tokio future. Without the feature they are left out and the link fails naming them. They are not stubbed out to abort at runtime instead.
• perry_ffi_spawn_blocking keeps its tokio-blocking-pool arm under async-runtime. Without it, the closure runs on turnloop's Occupancy::Long worker set (Pool has no long-occupancy job class: connection-lifetime work cannot move off tokio turnloop#42) through a new perry_runtime::turnloop_pool::submit_long, falling back to one plain thread (same worker_threads: a worker never runs module init (__perry_init_done_* is process-wide), so it aliases the spawning thread's heap — object literals read back property-less (OpenCode TUI wall) #10399 stack size) on a thread with no loop.
• perry_ffi_run_pending without tokio gives the whole budget to a bounded turn.

Features: crypto, bundled-bcrypt, bundled-argon2, compression-gzip, bundled-nodemailer, external-zlib-pump now imply async-bridge instead of async-runtime. Every feature that hands tokio a future keeps async-runtime: web-fetch, bundled-net, tls-runtime, external-tls-server, external-net-tls, bundled-ws, external-{net,ws,http-server,http-client}-pump, container. full still implies it, so prebuilt archives and PERRY_NO_AUTO_OPTIMIZE builds are unchanged.

CLI (optimized_libs/driver.rs): the unconditional force (and the UI force, and readline's mapping) becomes async-bridge. async-runtime is now selected for every shared-tokio wrapper through binding_needs_shared_tokio, the same predicate the #507 rebuild and the #7629 coherence check use, so the three cannot disagree. The existing pg/mysql2 and bundled-ws/bundled-net/http-client rules are kept.

turnloop#42 / alpha.8

PerryTS/turnloop#42 is closed. Occupancy::Long landed in turnloop #74 and first shipped in 0.1.0-alpha.5. Perry already pins alpha.6, so this PR does not need #11083's alpha.8 bump and is based on plain origin/main. submit_long keeps the existing Bounded path on the literal Loop::blocking call every P4 job already makes, and only the Long class goes through blocking_with.

Evidence (perrymaster, Node 26.5.1 from /opt/node-v26.5.1-linux-x64)

Both arms were built the same way from one shared target dir: cargo build --release -p perry -p perry-runtime-static -p perry-stdlib-static with CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16. Base is d8f24f15e (origin/main at branch point); the branch was later rebased onto 9d2693629 with no conflicts, then re-checked with cargo check and the gates below.

The subject: tokio leaves auto-optimized programs. Three fixtures (crypto async callbacks + WebCrypto, node:zlib routed to perry-ext-zlib, timer ordering) were compiled with auto-optimize on and run 3x each:

fixture	arm	auto-optimize features	tokio CGUs in stdlib .a	tokio-1. strings in binary	binary bytes	output vs node
crypto	base	async-runtime,crypto	16	43	12,022,336	same (3/3)
crypto	branch	async-bridge,crypto	0	0	11,874,696	same (3/3)
zlib (ext)	base	async-runtime,external-zlib-pump	16	43	12,046,816	same (3/3)
zlib (ext)	branch	async-bridge,external-zlib-pump	0	0	11,886,880	same (3/3)
timers	base	async-runtime	16	0 (dead-stripped)	9,210,784	same (6/6 on a quiet rerun)
timers	branch	async-bridge	0	0	9,210,784	same (6/6)

(The base timers binary diffed from Node on its first 3 runs while the host was compiling: t30 fired before t0. It matched 6/6 when rerun on a quiet host, and the branch binary also matched 6/6 then. Neither timers binary links tokio, and both are the same size, so this is a load-timing flake that exists on main.)

Shared-tokio programs still get tokio: the branch's auto-optimize archives for the net and ws/http gap tests were built with async-bridge,async-runtime,external-net-pump,… and each bundles 16 tokio CGUs.

Gap A/B: 53 tests, every test_gap_* matching ws / net_ / socket / crypto / zlib / bcrypt / argon2 / fetch / timer / readline / worker. Harness PERRY_SKIP_BUILD=1 ./run_parity_tests.sh --filter <name>, one test at a time. The ext-routed ones (ws, net, zlib) are auto-optimized by the harness itself.

• base: 53/53 PASS. branch: 53/53 PASS.
• Two things on the way are worth recording so nobody chases them:
  • With no npm ci in the tree, the 4 test_gap_turnloop_ws_* tests are PARITY_FAIL on both arms, because the Node oracle cannot resolve ws. The base and branch perry outputs were byte-identical. After npm ci --ignore-scripts both arms pass.
  • A first branch run had 11 COMPILE_FAILs. The cause was my own mid-run commit: auto-optimize archives stamped with a newer commit than the compiler binary gave "runtime library does not match this Perry compiler". After rebuilding the binary at HEAD and clearing the auto dirs, all 11 pass. It was not a code defect.

Unit tests (release, CGU16, RUST_TEST_THREADS=1):

• perry-runtime turnloop_pool: 11/11, including the new a_long_job_runs_while_every_bounded_worker_is_held. It holds every bounded worker on a gate (read from turnloop::pool_stats, not assumed) and requires the long job to complete anyway. Sabotage-checked: routing submit_long to Occupancy::Bounded makes it fail.
• perry-stdlib bridge tests (full features): 7/7. That covers the new reject_promise_later_rejects_with_the_message_on_the_next_pump and the three moved tokio tests, now in tokio_bridge::tests.
• perry-stdlib --no-default-features --features crypto,compression,email (tokio-free): new perry_ffi_async::tokio_free_tests::spawn_blocking_without_tokio_runs_off_thread_and_releases_its_inflight passes; 78 passed, 2 failed.
• The 2 failures are crypto::random::tests::native_dispatch_{pbkdf2,random_bytes}_value_form_fires_callback. They fail identically on base (base: 79 passed, 2 failed, same two), so they are pre-existing under --release.
• In the full-feature perry-stdlib --lib run, streams::pipe::tests::pipe_keeps_locks_until_async_abort_settles aborts the test binary on both base and branch. Also pre-existing.
• perry --bin perry -- optimized_libs stdlib_features shared_tokio: 63/63.

Checks and gates:

• RUSTFLAGS="-D warnings" cargo check -p perry-stdlib -p perry-runtime -p perry -p perry-ffi --all-targets (dev profile): clean. The tokio-free feature combinations async-bridge, crypto, compression-gzip, bundled-nodemailer, bundled-argon2, bundled-bcrypt, external-zlib-pump each cargo check clean, and cargo tree -i tokio finds no tokio in any of them.
• The lint job's script steps, run directly (run_lint_gates.sh is broken at its cargo-xwin step): cargo fmt --check, check_file_size.sh, tokio_inventory.py --self-test + check, gc_runtime_root_holders.py --self-test + check, unrooted_local_shape.py --check and --no-raise-vs, raw_handle_debt.py (both), addr_class_inventory.py, gc_pin_sites.py, check_cross_thread_promise_provenance.py, gc_root_dominance_check.py --audit-*, check_gc_scanner_latches.py, global_sink_isolation.py (both), lock_no_downgrade.py, check_test_registration.py, parity_known_failures.py --audit and 14 more: all 36 OK.

Inventory

python3 scripts/tokio_inventory.py: 17 edges before, 17 after. The perry-stdlib -> tokio manifest edge stays, because full and every tokio-socket feature still imply it. What moved is which programs reach it. The edge's surface / reached_when / blocker annotations are rewritten to say exactly that. source_sites.perry-stdlib is 73 -> 74 (the new file's header comment).

What remains for lane L (the whole edge)

With this split, the edge is deleted by removing tokio_bridge.rs, the three cfg(feature = "async-runtime") shims in perry_ffi_async.rs and dep:tokio, then folding async-runtime into async-bridge. Nothing in async_bridge.rs has to move. What still selects async-runtime, in the order it can go:

1. web-fetch: after deps(fetch): remove perry-stdlib's reqwest fallback (tokio group G) #11101 (lane G) lands, fetch has no tokio call sites. web-fetch then needs only async-bridge, and http-client can move to the bridge list in the driver. That is a one-line follow-up, and I deliberately did not stack on deps(fetch): remove perry-stdlib's reqwest fallback (tokio group G) #11101.
2. bundled-net, tls-runtime, external-tls-server, external-net-tls, bundled-ws: net/mod.rs, tls.rs and ws.rs still spawn tokio socket tasks. turnloop(P8 H): move perry-stdlib's TLS off tokio-rustls onto a sans-I/O rustls session #11102 (lane H) moves TLS off tokio-rustls but keeps tokio sockets, so P1 (turnloop handles under those three modules) is still needed.
3. external-{net,ws,http-server,http-client}-pump: perry-ext-net / -http hand tokio futures to perry_ffi_spawn_async and _with_reactor (lane A). perry-ext-net's upgradeTLS reply wait also calls Handle::current() inside perry_ffi_spawn_blocking.
4. pg / mysql2 / ioredis / mongodb (driver rule): their decline paths run Handle::current().block_on inside perry_ffi_spawn_blocking, which is only legal on tokio's own blocking pool (group B).
5. container: perry-container-compose is async tokio (group K).
6. Then the CLI's shared-tokio coherence guard (shared_tokio.rs, run_pipeline.rs, optimized_libs/driver.rs, no_auto.rs) goes dead and can be deleted. I left it alone here because tokio is not gone.

Not run

• The full gap sweep (it stalls under auto-optimize on perrymaster). Only the 53-test A/B above was run.
• cargo xwin check for Windows. This PR adds no cfg(windows) code.
• The public-baseline lint step (known red on main) and the benchmark/release-pipeline lint steps.
• No instruction-count A/B: nothing on a hot path changed, and the tokio-free programs simply do not link the runtime.
• cargo check -p perry-stdlib --no-default-features --features web-fetch fails on main as well (turnloop_client uses base64, which web-fetch does not enable). That is pre-existing, unrelated, and not touched here.
• Merge note: scripts/tokio_inventory.json's source_sites.perry-stdlib line is edited by this PR, deps(fetch): remove perry-stdlib's reqwest fallback (tokio group G) #11101 and turnloop(P8 H): move perry-stdlib's TLS off tokio-rustls onto a sans-I/O rustls session #11102. Whichever lands second resolves it by rerunning python3 scripts/tokio_inventory.py --update.

Summary by CodeRabbit

• New Features
  • Async promise handling and background tasks can now run without including the Tokio runtime.
  • Blocking work can use long-occupancy workers, with a thread fallback when those workers are unavailable.
• Improvements
  • Builds using supported libraries or UI features can produce smaller binaries without Tokio; features that require Tokio continue to include it.
  • Hashing and verification work now runs off the JavaScript event-loop thread.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a3936618-c5a8-4781-b13f-7639c087ff31

📥 Commits

Reviewing files that changed from the base of the PR and between 9d26936 and 70f9e66.

📒 Files selected for processing (18)

• changelog.d/11115-stdlib-async-bridge-without-tokio.md
• crates/perry-runtime/src/turnloop_pool/mod.rs
• crates/perry-runtime/src/turnloop_pool/tests.rs
• crates/perry-stdlib/Cargo.toml
• crates/perry-stdlib/src/argon2.rs
• crates/perry-stdlib/src/common/async_bridge.rs
• crates/perry-stdlib/src/common/mod.rs
• crates/perry-stdlib/src/common/tokio_bridge.rs
• crates/perry-stdlib/src/lib.rs
• crates/perry-stdlib/src/nodemailer.rs
• crates/perry-stdlib/src/perry_ffi_async.rs
• crates/perry-stdlib/src/readline/mod.rs
• crates/perry-stdlib/src/worker_threads/async_shim.rs
• crates/perry/src/commands/compile/link/build_and_run.rs
• crates/perry/src/commands/compile/optimized_libs/driver.rs
• crates/perry/src/commands/compile/optimized_libs/tests.rs
• crates/perry/src/commands/stdlib_features.rs
• scripts/tokio_inventory.json

Included review availability: Your plan provides up to 8 included reviews per hour; 3 remain after this review.

---

📝 Walkthrough

Walkthrough

The stdlib async bridge now compiles without Tokio, while Tokio runtime operations remain behind async-runtime. The compiler selects the bridge and runtime features separately. Blocking work can use turnloop’s Long occupancy pool when Tokio is absent.

Changes

Async Bridge and Blocking Work

Layer / File(s)	Summary
Long-occupancy submission crates/perry-runtime/src/turnloop_pool/mod.rs, crates/perry-runtime/src/turnloop_pool/tests.rs	The pool adds submit_long for Long occupancy while submit_rooted retains Bounded occupancy. A test checks that long work runs while bounded workers are occupied and that delivery occurs on the submitting thread.
Tokio-free promise bridge crates/perry-stdlib/Cargo.toml, crates/perry-stdlib/src/common/mod.rs, crates/perry-stdlib/src/common/async_bridge.rs, crates/perry-stdlib/src/argon2.rs, crates/perry-stdlib/src/nodemailer.rs, crates/perry-stdlib/src/lib.rs	The new async-bridge feature enables the promise bridge without Tokio. async-runtime now adds Tokio on top of that feature. Argon2 and nodemailer error paths queue promise rejections directly.
Tokio runtime separation crates/perry-stdlib/src/common/async_bridge.rs, crates/perry-stdlib/src/common/tokio_bridge.rs	Tokio runtime, spawn, and wait-driver operations move to tokio_bridge, which is compiled under async-runtime. The bridge installs its wait driver only when that feature is enabled.
Tokio-free blocking FFI path crates/perry-stdlib/src/perry_ffi_async.rs, crates/perry-runtime/src/turnloop_pool/mod.rs	Without async-runtime, perry_ffi_spawn_blocking submits work to Long occupancy workers, with OS-thread and inline fallbacks. The pending pump uses its budget for the pool turn. Tests check the no-Tokio execution path.
Feature selection and integration crates/perry/src/commands/compile/optimized_libs/driver.rs, crates/perry/src/commands/compile/optimized_libs/tests.rs, crates/perry/src/commands/stdlib_features.rs, crates/perry/src/commands/compile/link/build_and_run.rs, crates/perry-stdlib/src/readline/mod.rs, crates/perry-stdlib/src/worker_threads/async_shim.rs, scripts/tokio_inventory.json, changelog.d/11115-stdlib-async-bridge-without-tokio.md	The compiler selects async-bridge for features that do not require Tokio and retains async-runtime for Tokio-dependent features and shared-Tokio bindings. Readline and worker-thread feature gates use async-bridge. The inventory and changelog record the split.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant perry_ffi_async
  participant turnloop_pool
  participant LongWorker
  participant FallbackThread
  perry_ffi_async->>turnloop_pool: submit_long(work, deliver)
  turnloop_pool->>LongWorker: schedule work with Long occupancy
  perry_ffi_async->>FallbackThread: run work if submission is refused

Loading

Merge Risk: ⚪ Minimal · up to 70f9e

The promise bridge now builds without Tokio. Crypto, compression, readline, worker, and UI programs link smaller stdlib archives, while network and database features still pull in the Tokio runtime. No concrete breakage was identified, and the change looks ready to merge.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: separating the promise bridge from the Tokio runtime.
Description check	✅ Passed	The description is comprehensive and covers the change summary, implementation details, related work, test results, known failures, and remaining scope. It does not use every template heading or inclu…
Docstring Coverage	✅ Passed	Docstring coverage is 89.09% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 55 functions across 15 files. (3 skipped: 3…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}