fix(engine): preserve greedy sampling in mixed batches - #471
Open
taking-lying-flat wants to merge 1 commit into
Open
taking-lying-flat wants to merge 1 commit into
taking-lying-flat wants to merge 1 commit into
Conversation
taking-lying-flat
force-pushed
the
fix/greedy-mixed-batch
branch
from
September 14, 2026 16:57
d27f090 to
2fffff5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A greedy request with fixed logits
[1.0, 1.0, 0.0]returns token 0 alone, but can return token 1 when batched with a random request: the mixed path approximates greedy with temperature1e-6, leaving the tied maxima equally likely. Also,temperature=0, top_p=0.9unnecessarily enters probability sampling.Keep a per-request
greedy_maskand explicitly selectargmaxfor greedy rows. Treattemperature <= 0ortop_k == 1as greedy regardless oftop_p, and use neutral backend parameters for those rows. The all-greedy fast path is preserved. Mixed batches still run probability sampling across all rows; skipping that computation is outside this correctness fix.Related: #307. This fixes sampling-policy differences for identical logits; it does not establish or resolve the cause of the serial model-level divergence reported there.
Validation:
[1, 2, 2]instead of[0, 2, 1], and zero-temperature requests withtop_p < 1entered probability sampling. The validation file is retained locally and is not included in this PR.e0886ccplus this patch: 73 passed, 4 skipped, including real Triton CUDA sampling. The four FlashInfer cases were skipped because FlashInfer is not installed.git diff --checkpassed.Exact local validation command, from the repository root, using the untracked test file: