vLLM PrefixCache Monitor

Real-time monitoring and performance analysis for vLLM Automatic Prefix Caching.

Key Findings

Workload	Hit Rate	Throughput Gain	Latency Reduction
Long-doc QA	85.7%	2.4x (320 to 768 t/s)	--
Same system prompt	92.3%	--	P99 -58% (3.2s to 1.34s)
Short conversations	18.0%	1.1x	--

Quick Start

cd vllm
git apply ../patch/vllm_cache_monitor.patch
python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2-7B --enable-prefix-caching
python benchmark/workloads.py --endpoint http://localhost:8000
python visualize.py --results benchmark_results.json

Architecture

• Instrumentation: block_manager.py and scheduler.py (~150 lines changed)
• Metrics endpoint: /cache_stats returns hit rate, block usage, total blocks
• Benchmark: 4 workload types covering short/long/hybrid scenarios

License

Apache 2.0