Skip to content

README.md

Latest commit

 

History

History
31 lines (22 loc) · 969 Bytes

README.md

File metadata and controls

31 lines (22 loc) · 969 Bytes
Raw

vLLM PrefixCache Monitor

Real-time monitoring and performance analysis for vLLM Automatic Prefix Caching.

Key Findings

Workload Hit Rate Throughput Gain Latency Reduction
Long-doc QA 85.7% 2.4x (320 to 768 t/s) --
Same system prompt 92.3% -- P99 -58% (3.2s to 1.34s)
Short conversations 18.0% 1.1x --

Quick Start

cd vllm
git apply ../patch/vllm_cache_monitor.patch
python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2-7B --enable-prefix-caching
python benchmark/workloads.py --endpoint http://localhost:8000
python visualize.py --results benchmark_results.json

Architecture

  • Instrumentation: block_manager.py and scheduler.py (~150 lines changed)
  • Metrics endpoint: /cache_stats returns hit rate, block usage, total blocks
  • Benchmark: 4 workload types covering short/long/hybrid scenarios

License

Apache 2.0