Skip to content

Start runner: respawn a wedged isolate instead of hanging every request - #315

Draft
bittermandel wants to merge 2 commits into
plugin-host-revive-after-stallfrom
start-runner-wedge
Draft

bittermandel wants to merge 2 commits into
plugin-host-revive-after-stallfrom
start-runner-wedge

Conversation

@bittermandel

@bittermandel bittermandel commented Oct 6, 2026 •

Copy link
Copy Markdown

Stacked on #314.

Problem

StartEngine::handle calls the runner engine with no deadline. It also held the EngineSlot read lock for the whole call, so a pending maybe_respawn writer queued every later request behind it. If the runner's isolate thread stops scheduling (blocked in native code, or stuck in D on a page fault after a VM snapshot wake), every request that reaches the runner hangs for as long as the block lasts. The process stays alive, and nothing ever fails the request or replaces the engine.

Measured on a lynx dev orb with oj 0.2.16, freezing only the runner's engine thread (cgroup freezer):

t=0s   h3000=000/10.002670 mw=404
t=15s  h3000=000/10.002397 mw=404
...
t=105s h3000=000/10.002283 mw=404

Every :3000/health hits the curl timeout, while the plugin middleware answers instantly. This matches the dev-orb incident behind #314: after the wake, :3000 hung for more than 20 minutes while the respawned plugin middleware served /health 200 in 0.125 s. That orb is gone, so attributing its hang to this path is inferred.

Change

  • handle clones an Arc<EngineSlot> out of the lock, so a stuck call holds up neither a respawn nor other requests.
  • await_unless_wedged: while a runner call stays pending for OJ_START_UNRESPONSIVE (default 60 s), probe the same isolate with a trivial ping export. Calls interleave on the isolate, so a parked render still answers. Any reply means slow, so it keeps waiting. No reply, or Closed from an engine another request already retired, means wedged.
  • When wedged: abandon that engine first, so its blocked thread is never joined. Then swap in a fresh one (pointer-guarded, spaced by the window, at most 3 consecutive, reset when a request is served) and fail the request with a 500 that says whether the runner respawned.
  • OJ_START_UNRESPONSIVE is a non-scheduling limit: a single synchronous JS section longer than the window counts as wedged.

Tests (start_host.rs)

Test Wrong implementation it catches
a_slow_runner_call_is_awaited_not_declared_wedged A plain request timeout (fails the slow call)
a_natively_blocked_runner_call_is_declared_wedged No belt, i.e. plain call.await (hangs until the 10 s guard)
a_probe_on_a_retired_engine_counts_as_wedged Treating a locally failed Closed probe as a live isolate (waits forever on a stranded reply)

Each was mutation-checked against the wrong implementation named. cargo test -p oj 113 passed, -p oj_server 190, -p oj_env 10; clippy -D warnings and fmt clean.

Lynx RED → GREEN (same session, same scenario: only the runner's engine thread frozen for 240 s, :3000/health every 15 s with a 10 s client timeout)

  • RED, oj 0.2.16: every request hit the 10 s timeout for the whole freeze (t=0…105s h3000=000/10.0); the plugin middleware answered 404 instantly.
  • RED, aec2258 (first version of this PR, check inside the request future): still hung for 240 s, because each client disconnect at 10 s cancelled the check. Fixed in bba9cf9, which runs the check on its own task.
  • GREEN, bba9cf9: runner thread verified frozen throughout.
    t=0…105s  h3000=000/10.0      (frozen=1)
    t=120s    h3000=200/5.27      oj start: runner stopped scheduling; respawned its engine
    t=130…180s h3000=200/0.010    (old runner thread still frozen)
    

Not in this PR

  • ensure_runner_fresh ignores a pending runner_dirty when the runner bit is false. That's a pre-existing gap that a mode change across a respawn can expose; it needs its own repro.
  • No StartEngine-level test for maybe_respawn winning the race before wedge handling. There's no StartEngine harness; abandon-before-compare covers it by construction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant