perf: make server capture requests prefill-only

[ddlearn]

Motivation

Server-side spec capture only requires prefill and does not need to generate output tokens.

While profiling SGLang v0.5.14, I observed periodic decode batches accompanied by GPU utilization drops.

SGLang enables overlap scheduling by default. The scheduler may merge a completed prefill batch into running_batch before its CPU-side result processing marks the requests as finished.

With max_new_tokens=1, these requests use the normal generation path. Although prefill already produces the only requested token and the subsequent CPU processing marks the requests as finished, they remain in running_batch until the scheduler enters the decode scheduling path and calls update_running_batch().filter_batch().

When prefill requests keep arriving, the scheduler continues selecting prefill batches, so finished capture requests accumulate in running_batch. Once max_running_requests is reached, no more prefill requests can be admitted. The scheduler then enters the decode scheduling path:

• finished requests are filtered out;
• requests from the most recent prefill batch may still be marked unfinished because their CPU-side result processing is one step behind.

All required capture features are already produced during prefill, so this decode work is unnecessary and interrupts the prefill pipeline.

Setting max_new_tokens=0 uses SGLang's prefill-only path, which skips sampling and explicitly filters finished prefill-only requests from running_batch.

Modifications

• Change server-capture requests from max_new_tokens=1 to max_new_tokens=0.
• Use SGLang's prefill-only scheduling and cleanup path.
• Update the corresponding unit test expectation.

Related Issues

N/A.

Accuracy Test

Not applicable. This change does not modify model computation or the features captured during prefill.

The existing unit test is updated to verify:

{"temperature": 0.0, "max_new_tokens": 0}

Benchmark & Profiling

Profiling with max_new_tokens=1 showed finished capture requests accumulating in running_batch, followed by periodic unnecessary decode batches when max_running_requests was reached.

Using max_new_tokens=0 prevents this accumulation and removes the unnecessary decode phase. The short decode phase could not hide the overlapping CPU-side processing as effectively as prefill phase, exposing CPU latency and producing GPU idle gaps.

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://sgl-fru7574.slack.com/archives/C09784E3EN6 to discuss your PR.