Repository navigation
[diffusion] launch_server: accept pre-started scheduler workers - #8
Draft
OnePunchMonk wants to merge 1 commit into
Draft
OnePunchMonk wants to merge 1 commit into
OnePunchMonk wants to merge 1 commit into
Conversation
A scheduler worker spends most of its start-up importing torch, transformers, diffusers and sglang, and the server process imports the same stack before it spawns the workers, so the two imports run back to back. Callers can now start the workers first and register them with register_prespawned_schedulers(); launch_server then hands each worker its args instead of spawning a new process, and the imports overlap. Without registration nothing changes.
OnePunchMonk
added a commit
to OnePunchMonk/miles_diffusion
that referenced
this pull request
Oct 11, 2026
…mports sglang --rollout-prespawn-schedulers starts the scheduler workers in the sglang server process before its own sglang import, so the workers' import of the same stack runs in parallel instead of after it. Needs sglang's launch_server.register_prespawned_schedulers (OnePunchMonk/sglang#8); with an sglang that lacks it the waiting workers are told to exit and startup falls back to the old path. On H3 t2va (2x H200) engine boot went from about 138 s to 104 s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft in my fork for discussion with the miles-diffusion maintainers before proposing it upstream. Context: radixark/miles_diffusion#310 (startup time).
Problem: on an sglang-d engine start, the server process imports torch, transformers, diffusers and sglang (about 35 to 40 s), then spawns the scheduler workers, which import the same stack again (another 35 to 40 s). The two imports run back to back.
Change:
launch_server.register_prespawned_schedulers(factories)lets a caller start the workers first, before its own import, and register them.launch_serverthen hands each worker its args instead of spawning a new process. Without registration nothing changes.Result (MiniMax-H3 t2va, 2x H200, measured through miles-diffusion, two runs each): engine boot from server spawn to healthy went from about 138 s to 104 s; the worker import overlapped the server import almost fully (worker import 2.2 to 39.8 s, server import 1.0 to 40.4 s), and the scheduler bound the moment the server started instead of 31 s later.
Notes:
python/sglang/multimodal_gen/test/unit/test_launch_server_prespawn.py(2 passed).CI States
Latest PR Test (Base): ❌ Run #38160090073
Latest PR Test (Extra): ❌ Run #38160089869
Latest PR Test (AMD ROCm 10): ❌ Run #38160090009