Skip to content

fix(scheduler): match stop strings with incremental decoding - #464

Open
taking-lying-flat wants to merge 1 commit into
FlashML-org:mainfrom
taking-lying-flat:fix/stop-string-decoding
Open

taking-lying-flat wants to merge 1 commit into
FlashML-org:mainfrom
taking-lying-flat:fix/stop-string-decoding

Conversation

@taking-lying-flat

Copy link
Copy Markdown
Contributor

A stop string such as 🧬 can span four byte tokens despite having one Python character. The scheduler's character-sized token window could decode only part of it and let generation continue past the stop.

Reuse the frontend's incremental decode state for scheduler matching. Preserve the token context needed to finish decoding characters, and bound only the retained matching text. Keep both sides consistent when a frontend batch contains multiple messages for one request and when an ignored EOS participates in a matched stop. No new tokenizer dependency or model-specific handling is introduced.

Validation on Linux x86_64, Intel Core i7-12800HX, Python 3.12.13; CPU only, with no model checkpoint or GPU inference. Host GPU: NVIDIA RTX A1000 Laptop GPU (4 GiB), driver 595.84.

  • Local regression harness: 54 cases covering explicit UTF-8 byte tokens, Hugging Face ByteFallback/ByteLevel, WordPiece cleanup, Unicode stops, streaming clipping, EOS, and overlapping scheduler completion. Against base 08d728d5, 32 failed and 22 passed; all 54 pass with this change. The harness is retained locally and is not included in this PR.
  • Existing tokenizer, request, message, generation and prepare-stop accounting tests: 60 passed, 4 skipped because Qwen VL checkpoints were unavailable. Combined with the local harness: 114 passed, 4 skipped.
  • git diff --check passed.

Commands run from the source checkout:

PYTHONPATH=python /tmp/freetoken-glm-shared-venv/bin/python -m pytest /tmp/freetoken-stop-validation/test_stop_strings.py tests/tokenizer tests/scheduler/test_req_append_host.py tests/server/test_message_wire.py tests/server/test_generation_accounting.py -q --tb=short
PYTHONPATH=python timeout 30 /tmp/freetoken-glm-shared-venv/bin/python -m pytest tests/server/test_prepare_stop_accounting.py -q --tb=short -o faulthandler_timeout=10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant