Repository navigation
fix(limits): isolate in-flight Redis I/O on a dedicated thread and partition fallback - #2186
Conversation
…rtition fallback ## Problem 1. Main Tokio runtime starvation during request bursts: `ConnectionManager`'s socket pipeline task, claim `EVAL` deadlines, per-slot renewal tasks, release retries, and reconnect handshakes all ran on the serving Tokio runtime. On a CPU-capped replica (e.g. a 2-worker runtime), bursts of large JSON request bodies occupy the worker threads with synchronous parsing long enough that Redis replies sit unread past the 250 ms claim deadline — even when Redis itself answers in tens of microseconds. Three such timeouts drop the healthy connection and trigger a TLS reconnect on the same congested runtime, causing multi-second fallback windows. 2. Unpartitioned fallback and split counters: When falling back locally, each replica admitted up to the full global `limit` rather than its share across active replicas, did not count slots already admitted via Redis on that replica, and did not register fallback-admitted slots back into Redis once the connection recovered. ## Change - Dedicated `"inflight-redis"` OS thread: `SharedRedis::connect` spawns a dedicated OS thread with its own single-threaded Tokio runtime that owns `ConnectionManager` and executes all claims, 250 ms deadlines, withdrawals, release retries, reconnect loops, batched renewals, and pod heartbeats off the HTTP worker pool. Straggler timeouts/errors on an already-replaced connection no longer tear down the replacement connection. - Unified per-replica counter and scoped partitioned fallback: Every admitted slot on a replica (both Redis-admitted and fallback-admitted) is tracked in a single `PodInflightCounter`. Each replica lazily heartbeats only the limiter scopes it actually serves (`"realtime"` vs `"batch"`, 5 s interval / 15 s lease) so API replicas and batch daemon replicas do not inflate each other's replica counts, and local fallback caps concurrency at `limit.div_ceil(live_pods)`. - Batched renewals and reconnect re-registration: Replaces per-slot renewal tasks with a single 30 s batched `SYNC_KEY_SCRIPT` pass (chunked at 512 members per key). Active fallback slots are immediately synced to Redis upon reconnect so global counts stay accurate across failovers and recoveries; fallback slots that finish before ever reaching Redis drop with zero Redis traffic.
Deploying control-layer with
|
| Latest commit: |
546c74b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://410d5804.control-layer.pages.dev |
| Branch Preview URL: | https://fix-inflight-redis-thread.control-layer.pages.dev |
|
Running ultrareview automatically — this PR appears to change concurrency-sensitive code such as locks, queues, or retries, where a missed race may only surface under production load, so a deeper multi-pass review is worth running. I'll post findings when complete. |
There was a problem hiding this comment.
Ultrareview ran automatically — this PR appears to change concurrency-sensitive code such as locks, queues, or retries, where a missed race may only surface under production load, so a deeper multi-pass review is worth running.
2 issues found across 1 file
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="dwctl/src/realtime_inflight.rs">
<violation number="1" location="dwctl/src/realtime_inflight.rs:210">
P1: `state.connection` becomes visible before `sync_active_slots` finishes, so fresh claims can run while outage-held slots are still absent from Redis. Keep claims on fallback until the initial slot sync succeeds, then enable Redis claims.</violation>
<violation number="2" location="dwctl/src/realtime_inflight.rs:365">
P1: `div_ceil` gives every replica the remainder: with limit 1 and two live replicas, each pod gets a fallback limit of 1, admitting two slots globally. Assign the remainder to selected replicas, or use a conservative share, so fallback capacity does not exceed the configured limit.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| redis.connecting.store(false, Ordering::Release); | ||
| }; | ||
| let shared = Arc::new(connection); | ||
| state.connection.store(Some(Arc::clone(&shared))); |
There was a problem hiding this comment.
P1: state.connection becomes visible before sync_active_slots finishes, so fresh claims can run while outage-held slots are still absent from Redis. Keep claims on fallback until the initial slot sync succeeds, then enable Redis claims.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dwctl/src/realtime_inflight.rs, line 210:
<comment>`state.connection` becomes visible before `sync_active_slots` finishes, so fresh claims can run while outage-held slots are still absent from Redis. Keep claims on fallback until the initial slot sync succeeds, then enable Redis claims.</comment>
<file context>
@@ -92,48 +150,180 @@ impl SharedRedis {
- redis.connecting.store(false, Ordering::Release);
+ };
+ let shared = Arc::new(connection);
+ state.connection.store(Some(Arc::clone(&shared)));
+ state.connecting.store(false, Ordering::Release);
+ tokio::spawn(sync_active_slots(Arc::clone(&state), ConnectionManager::clone(&shared)));
</file context>
| } | ||
|
|
||
| fn acquire_fallback(&self, redis: &SharedRedis, account: &str, model: &str, key: String, limit: u32) -> Option<InflightSlot> { | ||
| let pod_limit = limit.div_ceil(redis.live_pods(self.scope)); |
There was a problem hiding this comment.
P1: div_ceil gives every replica the remainder: with limit 1 and two live replicas, each pod gets a fallback limit of 1, admitting two slots globally. Assign the remainder to selected replicas, or use a conservative share, so fallback capacity does not exceed the configured limit.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dwctl/src/realtime_inflight.rs, line 365:
<comment>`div_ceil` gives every replica the remainder: with limit 1 and two live replicas, each pod gets a fallback limit of 1, admitting two slots globally. Assign the remainder to selected replicas, or use a conservative share, so fallback capacity does not exceed the configured limit.</comment>
<file context>
@@ -153,206 +343,359 @@ impl RealtimeInflightLimiter {
+ }
+
+ fn acquire_fallback(&self, redis: &SharedRedis, account: &str, model: &str, key: String, limit: u32) -> Option<InflightSlot> {
+ let pod_limit = limit.div_ceil(redis.live_pods(self.scope));
+ let pod_guard = self.pod_counts.acquire(account, model, pod_limit)?;
+ let fallback_lease = redis.register_fallback_slot(self.scope, key);
</file context>
fergusfinn
left a comment
There was a problem hiding this comment.
Approved at 546c74bd.
The diagnosis matches production. dwctl_realtime_inflight_redis_fallbacks_total{scope="realtime"} came in one burst between about 01:00 and 09:00 UTC on 12 Nov: 35,067 fallbacks in the 05:00 hour, about 98,000 over the last 7 days, plus 699 Redis reconnects. The control-layer container's CFS throttled-period share peaked at 2.2% in that hour. So most of the lost 250 ms deadlines came from busy worker turns, not CPU quota, and timing the claim on its own current-thread runtime fixes that directly.
What I checked:
- Release racing the 30 s sync. A slot dropped between the snapshot and the
ZADDhits theactive_slots.get(id)→Nonebranch and is released. A drop that lands aftergetreturns waits on the DashMap shard lock, soin_redisis already true whenremovereads it. No DashMap reference is held across an.await. - A caller that goes away before the reply. The failed
reply.senddrops theSlotLease, which removes the slot and sends the release. - Reconnects. These only run on the worker runtime.
timed_outandreconnect_brokenignore stragglers from a connection that has already been replaced. - Fallback. It admits against
limit.div_ceil(live_pods)and counts Redis-admitted slots in the same per-pod counter. If Redis is lost entirely, the fleet now admits aboutlimitin total rather thanlimitper pod. The next sync backfills fallback slots, so the shared count includes them.
Two optional follow-ups, neither blocking:
reply_rx.awaithas no limit on the caller side. The worker always answers withinREDIS_TIMEOUTor drops the sender, so this works today. Wrapping it intimeout(RESPONSE_TIMEOUT, reply_rx)with a fallback would keep requests moving if that thread ever got stuck.PodInflightCounter::counterallocates both key strings on every call, including when the key already exists. That now happens on every Redis-admitted request, throughtrack.
Problem
ConnectionManager's socket pipeline task, claimEVALdeadlines, per-slot renewal tasks, release retries, and reconnect handshakes all ran on the serving Tokio runtime. On a CPU-capped replica (e.g. a 2-worker runtime), bursts of large JSON request bodies occupy the worker threads with synchronous parsing long enough that Redis replies sit unread past the 250 ms claim deadline — even when Redis itself answers in tens of microseconds. Three such timeouts drop the healthy connection and trigger a TLS reconnect on the same congested runtime, causing multi-second fallback windows.limitrather than its share across active replicas, did not count slots already admitted via Redis on that replica, and did not register fallback-admitted slots back into Redis once the connection recovered.Change
"inflight-redis"OS thread:SharedRedis::connectspawns a dedicated OS thread with its own single-threaded Tokio runtime that ownsConnectionManagerand executes all claims, 250 ms deadlines, withdrawals, release retries, reconnect loops, batched renewals, and pod heartbeats off the HTTP worker pool. Straggler timeouts/errors on an already-replaced connection no longer tear down the replacement connection.PodInflightCounter. Each replica lazily heartbeats only the limiter scopes it actually serves ("realtime"vs"batch", 5 s interval / 15 s lease) so API replicas and batch daemon replicas do not inflate each other's replica counts, and local fallback caps concurrency atlimit.div_ceil(live_pods).SYNC_KEY_SCRIPTpass (chunked at 512 members per key). Active fallback slots are immediately synced to Redis upon reconnect so global counts stay accurate across failovers and recoveries; fallback slots that finish before ever reaching Redis drop with zero Redis traffic.Tests
claims_complete_when_caller_runtime_workers_block: freezes all threads of a 2-worker caller Tokio runtime for2 × REDIS_TIMEOUTwhile a claim is in flight and verifies the dedicated thread completes the claim with 0 timeouts and 0 withdrawals.fallback_splits_the_limit_across_live_pods_per_scope: verifies"realtime"and"batch"heartbeats track live replicas independently and fallback divideslimitacross live replicas in the same scope.fallback_counts_slots_already_admitted_through_redis: verifies slots admitted via Redis count toward the replica's partitioned fallback cap when Redis disconnects.fallback_slots_sync_to_redis_on_reconnect_and_release_on_drop: verifies slots admitted locally during an outage sync into Redis upon reconnect and sendZREMwhen dropped.realtime_inflight::testsandcargo clippypass with zero warnings.