Skip to content

fix: Bound the wait for a procedure instance slot and report it - #5875

Open
Ludv1gL wants to merge 3 commits into
clockworklabs:masterfrom
Ludv1gL:fix/procedure-pool-wait-timeout
Open

fix: Bound the wait for a procedure instance slot and report it#5875
Ludv1gL wants to merge 3 commits into
clockworklabs:masterfrom
Ludv1gL:fix/procedure-pool-wait-timeout

Conversation

@Ludv1gL

@Ludv1gL Ludv1gL commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Description of Changes

ModuleInstanceManager::get_instance acquires a slot in the bounded procedure instance pool with a bare Semaphore::acquire_owned().await: no timeout and no bound on waiters. The pool defaults to one slot per core. Once every slot is held (a procedure that does outbound HTTP holds its slot for the whole call), every later procedure call on that database queues forever and the websocket caller never receives a result envelope, an error, or a timeout.

It is worse than it looks, because enqueue_procedure awaited the slot before returning and the websocket receive task (ws_recv_task) handles one message at a time. So a single queued procedure call also stalled every later message from that connection: subscribes, reducer calls and one-off queries. Only subscriptions that were already established kept flowing, since the send task is separate. From the client this looks like "the call never settles, subscriptions are fine".

Three commits, each reviewable on its own:

  1. Bound the wait. New procedure-queue-timeout in the [wasm] and [v8] config sections (default 30 seconds, 0 waits forever as before). When it elapses, get_instance fails with InstancePoolTimeout before anything was started, so the call is safe to retry. The error reaches websocket callers as a ProcedureStatus::InternalError result for their request id, HTTP /call and module HTTP handlers as 503, and the scheduler re-queues the item instead of dropping it.
  2. Wait off the receive loop. The checkout and the call are spawned once the call is validated, so the receive loop only pays for argument validation. Validation errors are still reported synchronously.
  3. Count timeouts. spacetime_procedure_instance_pool_timeouts_total{db, module_type}. The wait itself is already visible through spacetime_reducer_wait_time_sec and spacetime_worker_instance_operation_queue_length.

Not included, on purpose: a max-waiters cap (waiters are now bounded by timeout times arrival rate; a try_acquire fast-fail can be layered on if that ever matters), and a dedicated ProcedureStatus variant on the wire (the string status keeps this free of SDK codegen changes).

API and ABI breaking changes

None on the wire. ProcedureCallError and HttpHandlerCallError gain a PoolTimeout variant, and ModuleHost::call_procedure_with_params now returns PooledCallError instead of NoSuchModule. Both are host-internal types.

Expected complexity level and risk

  1. Small, localised change to the pool checkout and the websocket procedure path. The default timeout changes behaviour only for calls that would previously have hung for more than 30 seconds waiting for a slot.

Testing

  • New unit test bounded_pool_times_out_waiting_for_a_slot: a pool of one slot, held; a second checkout fails with InstancePoolTimeout; after the lease is returned, checkout succeeds.
  • options_from_partial_toml extended to cover procedure-queue-timeout as a duration string and as 0.
  • cargo clippy --all-targets clean on spacetimedb-core, spacetimedb-client-api, spacetimedb-standalone; cargo check --workspace --all-targets clean.

Verified locally

Standalone built from this branch, [wasm] procedure-instance-pool-size = 1, procedure-queue-timeout = "2s", a module with hold(secs) (a procedure that sleep_untils) and quick().

HTTP: with hold(8) in flight, quick() and hold(1) return 503 Timed out after 2s waiting for a free procedure instance; the call was not started after 2 s; the holder returns 8 normally.

One websocket connection sending hold(8), hold(8), quick(), Subscribe back to back:

before (master) after (this branch)
InitialSubscription +8.07 s (stuck behind the held slot) +0.07 s
second hold and quick silent until the slot freed InternalError result at +2.07 s
first hold Returned 8 Returned 8

`ModuleInstanceManager::get_instance` acquired a slot in the bounded
procedure pool with a bare `Semaphore::acquire_owned().await`. Once every
slot was held (a handful per database by default, one per core), every
later procedure call, HTTP handler call and scheduled procedure on that
database waited forever, and a websocket caller got no result envelope,
error or timeout at all.

Add `procedure-queue-timeout` to the `[wasm]` and `[v8]` config sections
(default 30 seconds, 0 waits forever as before). When it elapses,
`get_instance` fails with `InstancePoolTimeout` before anything was
started, so the caller can retry. The error reaches:

- websocket procedure calls as a `ProcedureStatus::InternalError` result
  for the request id, instead of silence
- HTTP `/call` and module HTTP handlers as 503 Service Unavailable
- the scheduler, which re-queues the item instead of dropping it

`enqueue_with_procedure_instance` now takes an already acquired lease so
the caller can report the timeout while it still owns the result target.
`enqueue_procedure` awaited the pool slot before returning, and the
websocket receive task handles one message at a time. With a full pool
this stalled every later message from that connection (subscribe,
reducer calls, one-off queries) until a slot freed up, or now until the
queue timeout elapsed. Only already established subscriptions kept
flowing, since the send task is separate.

Spawn the checkout and the call as a task once the call is validated, so
the receive loop only pays for argument validation. Validation errors
are still reported synchronously, and the result or the timeout error is
delivered to the same target as before.
Add `spacetime_procedure_instance_pool_timeouts_total`, labelled by
database and module type, so a saturated procedure pool shows up in
metrics rather than only as a stalled client.

The wait itself is already visible for WASM procedures through
`spacetime_reducer_wait_time_sec` (the call timer starts before the
checkout) and `spacetime_worker_instance_operation_queue_length`.
@Ludv1gL
Ludv1gL force-pushed the fix/procedure-pool-wait-timeout branch from 3f9e220 to 19abb54 Compare September 5, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant