Skip to content

fix(limits): isolate in-flight Redis I/O on a dedicated thread and partition fallback - #2186

Merged
rushilbhat merged 1 commit into
mainfrom
fix/inflight-redis-thread
Nov 15, 2026
Merged

rushilbhat merged 1 commit into
mainfrom
fix/inflight-redis-thread

Conversation

@rushilbhat

@rushilbhat rushilbhat commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  1. Main Tokio runtime starvation during request bursts: ConnectionManager's socket pipeline task, claim EVAL deadlines, per-slot renewal tasks, release retries, and reconnect handshakes all ran on the serving Tokio runtime. On a CPU-capped replica (e.g. a 2-worker runtime), bursts of large JSON request bodies occupy the worker threads with synchronous parsing long enough that Redis replies sit unread past the 250 ms claim deadline — even when Redis itself answers in tens of microseconds. Three such timeouts drop the healthy connection and trigger a TLS reconnect on the same congested runtime, causing multi-second fallback windows.
  2. Unpartitioned fallback and split counters: When falling back locally, each replica admitted up to the full global limit rather than its share across active replicas, did not count slots already admitted via Redis on that replica, and did not register fallback-admitted slots back into Redis once the connection recovered.

Change

  • Dedicated "inflight-redis" OS thread: SharedRedis::connect spawns a dedicated OS thread with its own single-threaded Tokio runtime that owns ConnectionManager and executes all claims, 250 ms deadlines, withdrawals, release retries, reconnect loops, batched renewals, and pod heartbeats off the HTTP worker pool. Straggler timeouts/errors on an already-replaced connection no longer tear down the replacement connection.
  • Unified per-replica counter and scoped partitioned fallback: Every admitted slot on a replica (both Redis-admitted and fallback-admitted) is tracked in a single PodInflightCounter. Each replica lazily heartbeats only the limiter scopes it actually serves ("realtime" vs "batch", 5 s interval / 15 s lease) so API replicas and batch daemon replicas do not inflate each other's replica counts, and local fallback caps concurrency at limit.div_ceil(live_pods).
  • Batched renewals and reconnect re-registration: Replaces per-slot renewal tasks with a single 30 s batched SYNC_KEY_SCRIPT pass (chunked at 512 members per key). Active fallback slots are immediately synced to Redis upon reconnect so global counts stay accurate across failovers and recoveries; fallback slots that finish before ever reaching Redis drop with zero Redis traffic.

Tests

  • claims_complete_when_caller_runtime_workers_block: freezes all threads of a 2-worker caller Tokio runtime for 2 × REDIS_TIMEOUT while a claim is in flight and verifies the dedicated thread completes the claim with 0 timeouts and 0 withdrawals.
  • fallback_splits_the_limit_across_live_pods_per_scope: verifies "realtime" and "batch" heartbeats track live replicas independently and fallback divides limit across live replicas in the same scope.
  • fallback_counts_slots_already_admitted_through_redis: verifies slots admitted via Redis count toward the replica's partitioned fallback cap when Redis disconnects.
  • fallback_slots_sync_to_redis_on_reconnect_and_release_on_drop: verifies slots admitted locally during an outage sync into Redis upon reconnect and send ZREM when dropped.
  • All 13 unit tests in realtime_inflight::tests and cargo clippy pass with zero warnings.

View guided diff Turn on auto-fix

…rtition fallback

## Problem

1. Main Tokio runtime starvation during request bursts: `ConnectionManager`'s
   socket pipeline task, claim `EVAL` deadlines, per-slot renewal tasks,
   release retries, and reconnect handshakes all ran on the serving Tokio
   runtime. On a CPU-capped replica (e.g. a 2-worker runtime), bursts of large
   JSON request bodies occupy the worker threads with synchronous parsing long
   enough that Redis replies sit unread past the 250 ms claim deadline — even
   when Redis itself answers in tens of microseconds. Three such timeouts drop
   the healthy connection and trigger a TLS reconnect on the same congested
   runtime, causing multi-second fallback windows.
2. Unpartitioned fallback and split counters: When falling back locally, each
   replica admitted up to the full global `limit` rather than its share across
   active replicas, did not count slots already admitted via Redis on that
   replica, and did not register fallback-admitted slots back into Redis once
   the connection recovered.

## Change

- Dedicated `"inflight-redis"` OS thread: `SharedRedis::connect` spawns a
  dedicated OS thread with its own single-threaded Tokio runtime that owns
  `ConnectionManager` and executes all claims, 250 ms deadlines, withdrawals,
  release retries, reconnect loops, batched renewals, and pod heartbeats off
  the HTTP worker pool. Straggler timeouts/errors on an already-replaced
  connection no longer tear down the replacement connection.
- Unified per-replica counter and scoped partitioned fallback: Every admitted
  slot on a replica (both Redis-admitted and fallback-admitted) is tracked in a
  single `PodInflightCounter`. Each replica lazily heartbeats only the limiter
  scopes it actually serves (`"realtime"` vs `"batch"`, 5 s interval / 15 s
  lease) so API replicas and batch daemon replicas do not inflate each other's
  replica counts, and local fallback caps concurrency at
  `limit.div_ceil(live_pods)`.
- Batched renewals and reconnect re-registration: Replaces per-slot renewal
  tasks with a single 30 s batched `SYNC_KEY_SCRIPT` pass (chunked at 512
  members per key). Active fallback slots are immediately synced to Redis upon
  reconnect so global counts stay accurate across failovers and recoveries;
  fallback slots that finish before ever reaching Redis drop with zero Redis
  traffic.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying control-layer with  Cloudflare Pages  Cloudflare Pages

Latest commit: 546c74b
Status: ✅  Deploy successful!
Preview URL: https://410d5804.control-layer.pages.dev
Branch Preview URL: https://fix-inflight-redis-thread.control-layer.pages.dev

View logs

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 14, 2026

Copy link
Copy Markdown

Running ultrareview automatically — this PR appears to change concurrency-sensitive code such as locks, queues, or retries, where a missed race may only surface under production load, so a deeper multi-pass review is worth running. I'll post findings when complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultrareview ran automatically — this PR appears to change concurrency-sensitive code such as locks, queues, or retries, where a missed race may only surface under production load, so a deeper multi-pass review is worth running.

2 issues found across 1 file

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="dwctl/src/realtime_inflight.rs">

<violation number="1" location="dwctl/src/realtime_inflight.rs:210">
P1: `state.connection` becomes visible before `sync_active_slots` finishes, so fresh claims can run while outage-held slots are still absent from Redis. Keep claims on fallback until the initial slot sync succeeds, then enable Redis claims.</violation>

<violation number="2" location="dwctl/src/realtime_inflight.rs:365">
P1: `div_ceil` gives every replica the remainder: with limit 1 and two live replicas, each pod gets a fallback limit of 1, admitting two slots globally. Assign the remainder to selected replicas, or use a conservative share, so fallback capacity does not exceed the configured limit.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

redis.connecting.store(false, Ordering::Release);
};
let shared = Arc::new(connection);
state.connection.store(Some(Arc::clone(&shared)));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: state.connection becomes visible before sync_active_slots finishes, so fresh claims can run while outage-held slots are still absent from Redis. Keep claims on fallback until the initial slot sync succeeds, then enable Redis claims.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dwctl/src/realtime_inflight.rs, line 210:

<comment>`state.connection` becomes visible before `sync_active_slots` finishes, so fresh claims can run while outage-held slots are still absent from Redis. Keep claims on fallback until the initial slot sync succeeds, then enable Redis claims.</comment>

<file context>
@@ -92,48 +150,180 @@ impl SharedRedis {
-            redis.connecting.store(false, Ordering::Release);
+            };
+            let shared = Arc::new(connection);
+            state.connection.store(Some(Arc::clone(&shared)));
+            state.connecting.store(false, Ordering::Release);
+            tokio::spawn(sync_active_slots(Arc::clone(&state), ConnectionManager::clone(&shared)));
</file context>

}

fn acquire_fallback(&self, redis: &SharedRedis, account: &str, model: &str, key: String, limit: u32) -> Option<InflightSlot> {
let pod_limit = limit.div_ceil(redis.live_pods(self.scope));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: div_ceil gives every replica the remainder: with limit 1 and two live replicas, each pod gets a fallback limit of 1, admitting two slots globally. Assign the remainder to selected replicas, or use a conservative share, so fallback capacity does not exceed the configured limit.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dwctl/src/realtime_inflight.rs, line 365:

<comment>`div_ceil` gives every replica the remainder: with limit 1 and two live replicas, each pod gets a fallback limit of 1, admitting two slots globally. Assign the remainder to selected replicas, or use a conservative share, so fallback capacity does not exceed the configured limit.</comment>

<file context>
@@ -153,206 +343,359 @@ impl RealtimeInflightLimiter {
+    }
+
+    fn acquire_fallback(&self, redis: &SharedRedis, account: &str, model: &str, key: String, limit: u32) -> Option<InflightSlot> {
+        let pod_limit = limit.div_ceil(redis.live_pods(self.scope));
+        let pod_guard = self.pod_counts.acquire(account, model, pod_limit)?;
+        let fallback_lease = redis.register_fallback_slot(self.scope, key);
</file context>

@fergusfinn fergusfinn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 546c74bd.

The diagnosis matches production. dwctl_realtime_inflight_redis_fallbacks_total{scope="realtime"} came in one burst between about 01:00 and 09:00 UTC on 12 Nov: 35,067 fallbacks in the 05:00 hour, about 98,000 over the last 7 days, plus 699 Redis reconnects. The control-layer container's CFS throttled-period share peaked at 2.2% in that hour. So most of the lost 250 ms deadlines came from busy worker turns, not CPU quota, and timing the claim on its own current-thread runtime fixes that directly.

What I checked:

  • Release racing the 30 s sync. A slot dropped between the snapshot and the ZADD hits the active_slots.get(id) → None branch and is released. A drop that lands after get returns waits on the DashMap shard lock, so in_redis is already true when remove reads it. No DashMap reference is held across an .await.
  • A caller that goes away before the reply. The failed reply.send drops the SlotLease, which removes the slot and sends the release.
  • Reconnects. These only run on the worker runtime. timed_out and reconnect_broken ignore stragglers from a connection that has already been replaced.
  • Fallback. It admits against limit.div_ceil(live_pods) and counts Redis-admitted slots in the same per-pod counter. If Redis is lost entirely, the fleet now admits about limit in total rather than limit per pod. The next sync backfills fallback slots, so the shared count includes them.

Two optional follow-ups, neither blocking:

  1. reply_rx.await has no limit on the caller side. The worker always answers within REDIS_TIMEOUT or drops the sender, so this works today. Wrapping it in timeout(RESPONSE_TIMEOUT, reply_rx) with a fallback would keep requests moving if that thread ever got stuck.
  2. PodInflightCounter::counter allocates both key strings on every call, including when the key already exists. That now happens on every Redis-admitted request, through track.

@rushilbhat
rushilbhat added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 8c7cfbb Nov 15, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants