Mistral's Voxtral Realtime delivers Whisper-quality speech recognition with sub-second streaming latency.
We introduce Voxtral Realtime, a natively streaming automatic speech recognition model that matches offline transcription quality at sub-second latency. Unlike approaches that adapt offline models through chunking or sliding windows, Voxtral Realtime is trained end-to-end for streaming, with explicit alignment between audio and text streams. Our architecture builds on the Delayed Streams Modeling framework, introducing a new causal audio encoder and Ada RMS-Norm for improved delay conditioning. We scale pretraining to a large-scale dataset spanning 13 languages. At a delay of 480ms, Voxtral Realtime achieves performance on par with Whisper, the most widely deployed offline transcription system. We release the model weights under the Apache 2.0 license.
Voxtral Realtime achieves offline-quality speech recognition at 480ms latency using native streaming architecture.
Delayed Streams Modeling with explicit audio-text alignment enables models to emit tokens based on acoustic evidence.
Causal encoder with modern architectural components outperforms adapted offline models for real-time transcription.
Streaming models cannot see future audio context, unlike offline systems that process entire signals.
Sub-second latency targets create severe model degradation when adapting traditional offline architectures.
Yuejie Li, Ke Yang +5
Spoken query retrieval is an important interaction mode in modern information retrieval. However, existing evaluation datasets are often limited to simple queries under constrained noise conditions, m...
Yitian Gong, Kuangwei Chen +10
Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretr...
Hangrui Hu, Xinfa Zhu +14
In this report, we present the Qwen3-TTS series, a family of advanced multilingual, controllable, robust, and streaming text-to-speech models. Qwen3-TTS supports state-of-the-art 3-second voice clonin...
Georgii Aparin, Tasnima Sadekova +6
Sparse Autoencoders (SAEs) are powerful tools for interpreting neural representations, yet their use in audio remains underexplored. We train SAEs across all encoder layers of Whisper and HuBERT, prov...
Xian Shi, Xiong Wang +11
In this report, we introduce Qwen3-ASR family, which includes two powerful all-in-one speech recognition models and a novel non-autoregressive speech forced alignment model. Qwen3-ASR-1.7B and Qwen3-A...
