Nemotron streaming

[cjpais]

We should support it and also see ggml

https://github.com/m1el/nemotron-asr.cpp

[cjpais]

probably use: https://github.com/altunenes/parakeet-rs/tree/master

[Leftium]

Something to consider when adding (Nemotron) streaming support: does transcribe-rs want to expose a single API that works for multiple streaming backends? (Nemotron, zipformer, local, cloud api, buffer/sentenced-based pseudo-streaming, etc)

Also consider these API's return a set of varied, rich results (not just strings/tokens), especially for streaming. (Timestamps, confidence, language, speaker diarization, interim results, etc)

parakeet-rs may be sufficient for basic Nemotron support, but the API seems very Nemotron-specific:

• assumes there is a 1:1 mapping between audio input chunks and transcription results
• assumes the transcription results don't ever change (no interim results)
• doesn't even seem to return timestamps; just strings

---

You can try out Nemotron local streaming transcription from https://rift-transcription.vercel.app

• There are instructions on how to download and start a local Nemotron WS server

Nemotron seems like a pretty solid streaming model.

[cjpais]

We will make our own abstraction, potentially dropping features of other backends. They will be exposed through existing methodologies in the codebase. Largely the focus of transcribe-rs is to be dead simple, however not necessarily feature complete. I am personally not interested in transcription results which change at the moment, though rewriting may happen in the future.

[chinarui-na]

It would be even better if the content could be displayed in real time while speaking and corrected after pauses

[cjpais]

we will start with the basics.

[Leftium]

It would be even better if the content could be displayed in real time while speaking and corrected after pauses

Nemotron's RNNT decoder is monotonic: tokens are never revised once emitted, so there's no "correction after pauses" at the decoder level. (Also other models can correct even before pauses!)

Since tokens are append-only and emitted per audio chunk, a simple chunk-in/text-out API like parakeet-rs delivers text at the same latency as a full streaming API. The difference is only metadata, which matters if you want richer UX but not for plain text insertion.

There are three independent axes of metadata for streaming results:

1. Stability: can future audio change this text? Always "no" for Nemotron (append-only). Relevant for models that revise earlier tokens (attention-based decoders, beam search).

2. Confidence: probability the model assigned to each token. The Nemotron model produces this, but parakeet-rs drops it. Also dropped by sherpa-onnx's NeMo decoder path.

3. Utterance boundary: segmentation that groups output into logical units (usually based on silence). Can be provided by the model, serving layer (e.g., sherpa-onnx endpoint detection), or the consuming application.

So this metadata can be produced/dropped at various stages: model (Nemotron), decoder (parakeet-rs, transcribe-rs, sherpa-onnx), and even app level.

If this data is available, better UX is possible like:

• Canceling transcriptions that started out poorly
• Cmd-Z undos the entire utterance at once, instead of single token/word at a time
• Just rendering interim text differently than finalized text
• Rendering confidence levels or highlighting words with low confidence