feat(kvcache): host-RAM KV tier for sparse-attention models (--kv-host-pages)

[alvarorsouza-arch]

What

Adds an opt-in second KV tier in pinned host RAM for models whose attention only re-reads a small selected subset of pages per step (Qwen3.8-Flash-Next QSA: indexer_budget=2048). The pinned mirror is the backing store; the GPU pool becomes an LRU cache of physical slots over the logical page space.

New flag: --kv-host-pages N (default 0 = off; zero behavior change when off).

Why this works for sparse attention

Every attend step first runs the indexer (top-k block selection over a compressed slab, fully GPU-resident) and only then reads the selected pages. Pages are fetched host→device after selection, bounded at ~34-520 pages/request/layer (fixed shape, CUDA-graph safe). Cold pages never cross PCIe; selected pages do, once.

Design

• Write-through mirror: every store_kv is mirrored to a pinned HostBank over UVA, so a GPU slot is always drop-clean; eviction is a pure rebind (no write-back, no dirty tracking)
• Residency engine: reuses the MoE offload machinery verbatim: flashlib lru_ensure (device-side LRU, fixed shapes, graph-safe) + fast_index_copy_multi_jit (one launch copies K+V of a page across all 12 QSA layers)
• Attention kernel untouched: after ensure, the block table is translated logical→physical; qsa_sparse_paged_attention runs unmodified
• Fixed-shape kernels everywhere: ensure_write_pages uses a single fixed-shape Triton launch (stride samples + one boundary sample per request run). The first version used boolean-mask indexing and torch.unique, both data-dependent shapes that hide device-to-host syncs per layer and serialize the prefill-overlap pipeline (the same pattern later found in moe/offload_cache.py)
• Anti-silent-corruption: compact_selected_pages gained a drop counter; max_sel_pages sized from 2*block_topk + 8 instead of the contiguous-selection estimate

Measured (RTX 4070 Ti SUPER 16 GB, EPYC 7K62, 251 GB RAM, PCIe gen4)

Config A (fp8 KV via #354 composed): --memory-ratio 0.92 --max-prefill-length 4096 --moe-cache-size 1024 --ple-backend pinned --max-running-requests 1 --enable-special-token-ckpt --kv-cache-dtype fp8 --kv-host-pages 5000 --num-pages 1635

Metric	Value
Logical KV capacity	424,640 tokens (1,635 GPU-resident + 5,000 host pages)
Needle @ 195,486-token context	pass (secret code retrieved)
Cached-prefix turn (4,617 new tokens over 67,818 cached)	3.3-3.4 s end-to-end
Prefill, cold 4096-token chunk	~1,238 t/s
Decode at 195k context	~26.5 t/s

Composes with #354 (fp8 KV): fp8 pages halve fetch bytes and double the resident pool.

Known limitations

• Decode degrades when the selection working set outgrows the physical pool (LRU thrash); pool size sets the cliff, fp8 halves the per-miss cost
• Only QSA (qwen4_exp) is wired; the same pattern applies to dsv4_sparse
• A single request is capped by the checkpoint's rope table (262,144 for Qwen3.8-Flash-Next); extra pool capacity serves concurrent requests and the prefix cache

Files

• New: kvcache/kv_host_offload.py, kernel/triton/qsa/offload.py
• Modified: attention/qsa_sparse.py, kvcache/qsa_pool.py, kvcache/__init__.py, core.py, engine/config.py, engine/engine.py, server/args.py

Test protocol

All numbers measured end-to-end over the OpenAI API. The quality suite (40 known-answer QA + 30 cloze at temperature 0) scored identical to the non-offload baseline on the same server. Happy to add unit tests for the offloader where maintainers prefer.