Conversation
Every distinct (batch,seq) ShapePlan allocates a persistent arena plus up to 64 MiB of cuBLASLt workspace, a cudaGraph and a cudaGraphExec, retained for the worker's lifetime. A long-lived worker embedding diverse batch shapes therefore accumulated one arena per shape and could OOM the 6 GiB RTX 4050 (observed out-of-memory crashes in the live gateway log at cuda_family_common.cuh:73). Retain at most two warm plans per context. Eviction happens before allocation of a new plan and after cudaStreamSynchronize, so a victim's buffers are freed while no kernel references them; the just-executed plan is always promoted to most-recently-used and never the victim. Deleting a plan destroys its DeviceAllocation members (cudaFree) via destructors, so eviction is what bounds VRAM. Also restores the plans member declaration that the eviction edit had dropped (build previously failed with 7 undefined-identifier errors). Verified live: four distinct shapes captured, replay of the evicted first shape bit-identical (max_abs_delta=0), steady serving VRAM ~1.9 GiB on the RTX 4050 lane.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
![synapse-alfonso[bot]](https://avatars.githubusercontent.com/u/271195168?s=60&v=4){kind=small}
There was a problem hiding this comment.
Confirmed the defect on master: plans is unbounded, keyed by (batch, seq), and each entry holds a captured graph plus a full arena, so variable-length traffic grows VRAM without a ceiling. The fix is the right shape, and the ordering argument in the comments holds — run() returns after cudaStreamSynchronize, so an eviction after it frees nothing in flight.
One defect in the fix, reachable on the failure path. The map and the LRU list are updated at different times: emplace at insert, touch_plan only after run() returns. forward catches and returns -1 on a throw, so the process survives, and after a run() throw the plan is in plans but never in plan_lru. Consequences:
- One throw: that plan becomes a zombie — never touched, never a victim — and the cache holds 1 live entry plus a permanent arena. The bound is silently 3, not 2.
- Two consecutive throws on distinct new shapes:
plans.size() == 2withplan_lruempty, and the next miss callsplan_lru.back()on an empty list, which is undefined behaviour rather than an error.
Both go away if the list and the map are kept in step: call touch_plan(key) immediately after emplace (before run), then again after run as you do now. Every entry in plans is then always in plan_lru, the eviction victim is always a real key, and the bound holds regardless of what run() does. That also makes the plans.find(key) == end() guard in touch_plan unreachable in practice, which is fine to keep as a defensive check.
One question, not a blocker: is 2 the right constant for the module's traffic? A miss costs initialize_and_verify, which runs a full verification forward, so thrash is expensive. Interactive queries and background batches on this lane tend to be exactly two shapes, so 2 is probably right today — but it is worth a short comment saying why 2, so the next person who sees thrash in the perf log knows which number to move.
Happy to merge on the touch-on-insert change; I will re-verify on a rented 4090 alongside #17/#18 rather than three separate rentals.
There was a problem hiding this comment.
That is the fix: touch on insert, touch after run retained for hit promotion, so every entry in plans is always in plan_lru and the miss-path .back() is always a real key. Verified by reading the resulting forward body rather than the diff.
Merging after a hardware pass that I have folded into the rental already running for #17/#18: four shapes in sequence against a pr-16 worker to observe the eviction and the rebuild, with the rebuilt plan's output byte-compared against the first forward and against a master-built worker. The LRU changes retention, not arithmetic, so byte-identity is the expected result; if a rebuilt plan produced different bytes that would be the finding.
Problem
QwenContextretains every distinct(batch, seq)ShapePlanfor the worker's lifetime. Each plan owns a persistent arena (scales withbatch*seq*hidden + batch*heads*seq^2, see #2314-style arithmetic), up to 64 MiB of cuBLASLt workspace, acudaGraphand acudaGraphExec. A long-lived worker that embeds diverse batch shapes therefore accumulates one arena per distinct shape.Observed live: the production gateway serving AFT/MC traffic logged repeated
cudaMalloc ... out of memoryengine crashes (cuda_family_common.cuh:73) after accumulating shapes like64x172(arena 646 MB),16x1117(1.59 GB), and dozens of 1xN shapes — on a 6 GiB RTX 4050, where persistent f16 weights alone are ~1.1 GiB.Fix
Bound the cache: retain at most two warm plans per context (
max_plans = 2, LRU order).cudaStreamSynchronizeon the context's single stream, so victim buffers are freed while no kernel references them; the just-executed plan is promoted to MRU and never the victim.unique_ptr<ShapePlan>, whose destructor chain frees graph exec, graph, and everyDeviceAllocation(cudaFree). Eviction is what bounds VRAM.plansmember declaration that the eviction edit had dropped (the tree previously failed to compile with 7 undefined-identifier errors).Verification (live RTX 4050, production gateway lane)
synapse-worker-cuda(--features cuda) green.1x34,1x66,1x98,1x130) each captured a plan (captured_exact=truein worker logs).0.0), norms all 1.0 — correctness preserved across eviction.Notes
max_plansif a workload shows pathological eviction thrash.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by cubic
Bounds the Qwen3 shape-plan cache to two entries with LRU eviction, preventing VRAM exhaustion on workers that see many distinct batch shapes.
run()executes, so a failed run can't orphan an arena.plansmember declaration that was accidentally dropped, fixing a compile error.Written for commit 5b6685b. Summary will update on new commits.