Repository navigation
vllm: serve AWQ weights with MTP and record the benchmarks - #1011
Merged
Merged
Conversation
psi pushes the llama-swap health result again, but Gatus only accepts pushes for endpoints it declares, so every push failed with 404 and the check unit kept failing.
llama-swap looks up nvidia-smi on PATH for its web UI GPU panel; without it the panel stays empty and startup logs a misleading 'GPU monitoring not available'.
llama-swap only forwards its own proxy log to stdout by default, so vLLM's startup report (KV capacity, load failures) never reached the journal, and the web UI log view had no timestamps to correlate cold starts with requests.
Each on-demand start ran in a fresh container, so torch.compile, FlashInfer and Triton output was rebuilt every time even though vLLM already writes reusable AOT artifacts; keeping them on the host shortens every cold start after the first.
The choice of weights, KV format, speculative decoding and engine rests on long evaluation rounds whose results otherwise live only in a local harness. Keep them next to the module, and record that default.nix does not yet apply those decisions.
FP8 weights leave only 689k KV tokens, about 2.6 requests at the 262k context limit, which is too little for ten users sharing the server. AWQ was non-inferior on long-context accuracy and tool calling, and MTP gave the best multi-user throughput, so together they reach 1.53M tokens without an accuracy cost.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The FP8 weights leave only 689k KV tokens, about 2.6 requests at the 262k context limit, which is too little for the roughly ten lab users the planning agent will bring. This moves llama-swap's vLLM to the AWQ weights with MTP speculative decoding, reaching 1.53M KV tokens: long-context accuracy and tool calling were non-inferior to FP8 in pai red benchmarks, and MTP gave the best multi-user throughput of the six modes tested. SGLang was also compared and rejected for now because its cache reuse collapses at 16 concurrent users on different documents. The full results live in
modules/vllm/README.md, alongside the operational follow-ups from running the service: the gatus endpoint, nvidia-smi for the web UI, upstream logs with timestamps, and JIT caches that persist across cold starts.nix evalconfirms psi's final command, and the same flags ran in the benchmark harness, but this exact service has not been started through the production path yet. The first start after deploy downloads about 24 GB of weights, so the first request will wait.