VibeVoice-ASR-BitNet Shrinks Speech Recognition to 1.58 GB With Heterogeneous Quantization, Enabling Real-Time Transcription on Three CPU Threads

Microsoft Research has released the VibeVoice-ASR-BitNet technical report, MIT-licensed weights, and a C++ inference implementation. The project aims to run language-model-based multilingual speech recognition on standard x86 or ARM CPUs. It retains VibeVoice-ASR’s “VAE tokenizer plus autoregressive decoder” architecture, but replaces the original Qwen2.5-7B decoder with the 1.5B version to reduce memory requirements on edge devices.

Rather than applying a single format to every component, the compression scheme is heterogeneous. The team found that the ConvNeXt VAE tokenizer is primarily constrained by activations and memory transfers, so it uses I8_S throughout the pipeline and fuses normalization, convolution, and activation functions to reduce reads and writes of intermediate tensors. The Transformer decoder, which is dominated by weight bandwidth, instead uses I2_S ternary weights, while embeddings and a small number of sensitive components remain in Q6_K. This reduces the model’s total size from 4.62 GB in FP16 to 1.58 GB, a compression ratio of approximately 2.9×.

In official tests on an AMD EPYC 7V13 using 20 seconds of audio, the model achieved a real-time factor of 0.77 with three threads, meaning it processed audio faster than playback time. At a similar size of roughly 1.6 GB, its end-to-end speed was 1.55× to 2.28× faster than INT8 Whisper.cpp large-v3-turbo. The team has also released AVX2 and NEON SIMD kernels and GGUF files that can be loaded through llama.cpp, Ollama, or Docker Model Runner.

The trade-off is that accuracy does not remain uniformly on par with the baseline. Compared with the 7B baseline, WER or CER increases across several datasets; on the Chinese AISHELL-4 dataset, CER rises from 19.83% to 27.45%. Moreover, “real time” currently refers only to the speed of processing complete audio segments: the implementation does not yet support streaming chunks or incremental decoding. Engineering teams should therefore evaluate performance on their actual CPUs and with relevant accents, noise conditions, and long recordings rather than basing deployment decisions on a single EPYC benchmark.