add streaming beam search for cache aware models to NeMo inference

[lilithgrigoryan]

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Adds MALSD beam search to cache-aware RNNT streaming inference, including K-beam hypothesis carry-over across chunks, cumulative hypothesis publishing, per-stream phrase boosting, and optional n-gram LM fusion.

In beam mode, a single stream may contain multiple utterances, and hypothesis state is managed as follows:

1. Between chunks: keep the top K hypotheses active and publish the current best hypothesis as the partial transcript.
2. At EOU / pause detection: finalize the best beam for the current utterance, commit it to the cumulative transcript, clear only the in-flight suffix, and continue processing the same stream. Decoder state is preserved so the next utterance can continue in context.
3. At stream end: clear all beam and transcript state for that stream, matching greedy-mode cleanup behavior.

Endpointing behavior is unchanged and still relies on the existing silence/VAD logic. Beam mode only changes what happens when a segment ends: the best beam is selected and committed as the finalized utterance.

Collection: asr (inference pipelines, model wrappers, streaming state, decoding)

Changelog

• Add specific line by line info of high level changes in this PR.

Usage

Example usage during N-Gram LM fusion:

python examples/asr/asr_streaming_inference/asr_streaming_infer.py \
  --config-path=../conf/asr_streaming_inference \
  --config-name=cache_aware_rnnt \
  asr.decoding.strategy=malsd_batch \
  asr.decoding.beam.beam_size=4 \
  asr.decoding.beam.ngram_lm_model=/path/to/lm.nemo \
  asr.decoding.beam.ngram_lm_alpha=0.5 \
  asr.decoding.beam.enable_per_stream_biasing=true \
  audio_file=/path/to/manifest.json

Cache-aware streaming WER — Riva word-boosting eval

Model: nvidia/nemotron-speech-streaming-en-0.6b
Manifest: internal set
Word boosting: internal
Batch size: 256
Metric: Whisper-normalized WER (%)
Beam search: MALSD, beam size = 4 (malsd_batch)

WER (%) — lower is better

Decoder	[70, 13]	[70, 6]	[70, 1]	[70, 0]
Greedy	25.54	26.50	29.57	32.80
Greedy + WB	22.40	23.74	26.92	30.30
MALSD (b=4)	24.98	25.89	29.07	32.77
MALSD + WB	18.70	20.05	23.78	28.27
Δ WB (greedy)	−3.14	−2.76	−2.65	−2.50
Δ WB (beam)	−6.28	−5.84	−5.29	−4.50

RTFx — higher is faster

Decoder	[70, 13]	[70, 6]	[70, 1]	[70, 0]
Greedy	546.40	394.50	179.26	109.63
Greedy + WB	522.08	390.60	177.78	108.94
MALSD (b=4)	512.49	349.99	141.53	77.91
MALSD + WB	474.22	330.63	133.34	74.56

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[github-actions]

[🤖]: Hi @lilithgrigoryan 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

[naymaraq]

This is really great work. Thanks @lilithgrigoryan

I’ll do another round of review after this one, since I see the PR is still a work in progress.

[naymaraq]

For consistency, please bring back the argument descriptions in the docstring.

[lilithgrigoryan]

I believe, I kept the original docstring for execute_step unchanged. Git’s diff makes the execute_step and encoder_step (a new function) docstrings look confusing for some reason.

LMK, if I miss something.

[naymaraq]

Argument descriptions are missing for encoder_step

[github-actions]

[🤖]: Hi @lilithgrigoryan 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

[github-actions]

[🤖]: Hi @lilithgrigoryan 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

[naymaraq]

I assume, 2025 should be 2026

[naymaraq]

The error should be raised as early as possible, before the model is loaded.

[naymaraq]

The rest looks good to me! Just 2–3 minor comments.

Ideally, for completeness, you could also add a comparison of greedy decoding, beam search, and beam search with nGPULM, but that's up to you.

[naymaraq]

LGTM! Great work. Thanks @lilithgrigoryan !
Just make sure the tests are passing; otherwise, I don’t have any other comments.