Repository navigation
Start runner: respawn a wedged isolate instead of hanging every request - #315
Draft
bittermandel wants to merge 2 commits into
Draft
bittermandel wants to merge 2 commits into
bittermandel wants to merge 2 commits into
Conversation
…awned instead of hanging every request
…client cannot cancel it
bittermandel
force-pushed
the
start-runner-wedge
branch
from
October 6, 2026 10:34
bba9cf9 to
c83ab4f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #314.
Problem
StartEngine::handlecalls the runner engine with no deadline. It also held theEngineSlotread lock for the whole call, so a pendingmaybe_respawnwriter queued every later request behind it. If the runner's isolate thread stops scheduling (blocked in native code, or stuck in D on a page fault after a VM snapshot wake), every request that reaches the runner hangs for as long as the block lasts. The process stays alive, and nothing ever fails the request or replaces the engine.Measured on a lynx dev orb with oj 0.2.16, freezing only the runner's engine thread (cgroup freezer):
Every
:3000/healthhits the curl timeout, while the plugin middleware answers instantly. This matches the dev-orb incident behind #314: after the wake,:3000hung for more than 20 minutes while the respawned plugin middleware served/health200 in 0.125 s. That orb is gone, so attributing its hang to this path is inferred.Change
handleclones anArc<EngineSlot>out of the lock, so a stuck call holds up neither a respawn nor other requests.await_unless_wedged: while a runner call stays pending forOJ_START_UNRESPONSIVE(default 60 s), probe the same isolate with a trivialpingexport. Calls interleave on the isolate, so a parked render still answers. Any reply means slow, so it keeps waiting. No reply, orClosedfrom an engine another request already retired, means wedged.OJ_START_UNRESPONSIVEis a non-scheduling limit: a single synchronous JS section longer than the window counts as wedged.Tests (start_host.rs)
a_slow_runner_call_is_awaited_not_declared_wedgeda_natively_blocked_runner_call_is_declared_wedgedcall.await(hangs until the 10 s guard)a_probe_on_a_retired_engine_counts_as_wedgedClosedprobe as a live isolate (waits forever on a stranded reply)Each was mutation-checked against the wrong implementation named.
cargo test -p oj113 passed,-p oj_server190,-p oj_env10; clippy-D warningsand fmt clean.Lynx RED → GREEN (same session, same scenario: only the runner's engine thread frozen for 240 s,
:3000/healthevery 15 s with a 10 s client timeout)t=0…105s h3000=000/10.0); the plugin middleware answered 404 instantly.Not in this PR
ensure_runner_freshignores a pendingrunner_dirtywhen the runner bit is false. That's a pre-existing gap that a mode change across a respawn can expose; it needs its own repro.maybe_respawnwinning the race before wedge handling. There's no StartEngine harness; abandon-before-compare covers it by construction.