Repository navigation
Megakernel backend for Nemotron-TTS - #68
Closed
adichatterje wants to merge 3 commits into
Closed
adichatterje wants to merge 3 commits into
adichatterje wants to merge 3 commits into
Conversation
adichatterje
requested review from
anand-nv,
pskrunner14 and
rmittal-github
as code owners
October 15, 2026 11:52
Contributor
|
Important Review skippedToo many files! This PR contains 639 files, which is 339 over the limit of 300. To get a review, reduce the PR to 300 files or fewer by splitting it into smaller PRs or changing its base branch. ⚙️ Run configuration
⛔ Files ignored due to path filters: 1
📒 Files selected for processing: 639
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
 [High impact] Adds Nemotron native TTS backend with CUDA kernels and model conversion. Fix C API creation and the dropped CLI runtime settings before merging. FindingsSummaryAdds an optional H100 Nemotron-TTS backend with CUDA kernels, native text preparation, Python batching, and C++ service integration.
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart LR
Client[CLI / C API / HTTP / gRPC] --> Synth[Synthesizer]
Synth --> Magpie[Existing Magpie runtime]
Synth --> Native[Native runtime and text preparation]
Native --> Socket[Unix worker connection]
Socket --> Batch[Fixed cohort or in-flight batcher]
Batch --> GPU[CUDA prefill, stage 0 and codec]
GPU --> PCM[Streamed PCM]
PCM --> Client
Reviews (1) · Last reviewed commit: 721c67d · Reviewed by Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+220
to
+223
| if (HAS_FIELD(model, nemo_speech_tts_model_config, model_backend) && model->model_backend) | ||
| cfg.backend = model->model_backend; | ||
| if (HAS_FIELD(model, nemo_speech_tts_model_config, native_socket)) | ||
| cfg.native_socket = str_or_empty(model->native_socket); |
There was a problem hiding this comment.
A C caller that sets model_backend="nemotron" and native_socket, but leaves magpie_model and codec_model empty, still fails in nemo_speech_tts_create before connecting to the worker. Creation unconditionally requires both Magpie paths even though Nemotron does not use them. Apply those checks only to the Magpie backend.
Knowledge Base Used: Text-to-speech synthesis
Comment on lines
+65
to
+72
| if (parsed.backend == "nemotron") { | ||
| nemo_speech::tts::SynthesizerConfig config; | ||
| config.backend = parsed.backend; | ||
| config.native_socket = parsed.native_socket; | ||
| config.native_timeout_seconds = parsed.native_timeout_seconds; | ||
| config.default_language_code = parsed.default_language_code; | ||
| config.default_voice_name = parsed.default_voice_name; | ||
| return config; |
There was a problem hiding this comment.
The Nemotron branch drops parsed.runtime. For example, synthesize "Hello" --tts-backend nemotron --native-socket /tmp/nemotron.sock --tts.seed 42 accepts the setting but sends seed 0, because the new configuration keeps the default seed. Benchmarks reuse this helper and can report a seed that the worker did not use. Copy parsed.runtime into config.runtime before returning, as the serving path already does.
Suggested change
| if (parsed.backend == "nemotron") { | |
| nemo_speech::tts::SynthesizerConfig config; | |
| config.backend = parsed.backend; | |
| config.native_socket = parsed.native_socket; | |
| config.native_timeout_seconds = parsed.native_timeout_seconds; | |
| config.default_language_code = parsed.default_language_code; | |
| config.default_voice_name = parsed.default_voice_name; | |
| return config; | |
| if (parsed.backend == "nemotron") { | |
| nemo_speech::tts::SynthesizerConfig config; | |
| config.backend = parsed.backend; | |
| config.runtime = parsed.runtime; | |
| config.native_socket = parsed.native_socket; | |
| config.native_timeout_seconds = parsed.native_timeout_seconds; | |
| config.default_language_code = parsed.default_language_code; | |
| config.default_voice_name = parsed.default_voice_name; | |
| return config; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Nemotron-TTS megakernel backend optimized for H100 for TTFA p95
This allows us to support up to batch size 512 within 200 ms TTFA p95.
Build and serving instructions: https://github.com/adichatterje/NeMo-Speech.cpp/blob/adichatterje/megakernel/nemotron-tts/docs/nemotron-megakernel.md#build
Measurements: https://docs.google.com/spreadsheets/d/1U0g30y94x6r5EaXxKM9SUn_YgL-NsZ2bcdHXGfzBC1k/edit?gid=326225413#gid=326225413
Output audios: https://drive.google.com/drive/folders/18dsr7XCHzZOfurTntVqwaLEC5Y5XbHVN
Jira: https://nvidia.atlassian.net/browse/RIVA-8275