AI 研究
Kyutai’s AURA Generates Streaming Avatars from Speech, with Caveats for Single-GPU Real-Time Use
AURA does not need to receive several seconds of future audio in advance. But the public reference implementation is slower than the paper’s reported results, and the dataset has not been fully released, so deployment and reproducibility should be assessed separately.
To give a voice assistant a speaking face, developers need video to be generated as audio arrives. Kyutai’s AURA takes a portrait and speech as input and can begin generating corresponding frames using only the current audio segment, reducing the need to wait for several seconds of future speech. It handles talking-head avatars; other models are still needed to understand questions and produce answers. The research paper was submitted on November 19, and Kyutai published an introduction the following day. Paper abstract, official introduction
Segment-by-segment generation reduces the wait for future audio
AURA distills a generative model that originally processed longer videos at once into a model that needs only four denoising steps per segment. Denoising is the process of gradually turning noise into an image. AURA also uses a rolling KV cache to retain attention-computation information from previous frames, allowing new segments to draw on recent motion context while keeping the cache size under control. This design lets the model continuously receive audio and output video segment by segment, but it still has to wait for the current segment and its computation to finish; it should not be considered zero-latency. Methods section
The paper’s real-time speed differs from the public implementation
The authors measured performance on a single H100 using a fixed ten-second English audio clip and a 960×960 reference image. They report that the 1.3 billion-parameter version reaches 20.7 frames per second, above the 16 fps playback requirement. The 5 billion-parameter version reaches 9.7 fps at 720p, below its 25 fps playback rate. The end-to-end measurement includes relevant overhead but excludes loading and warm-up. These are results reported by the team and do not establish that a typical consumer GPU can achieve the same performance. Experimental setup and results
The model card is a closer guide to the download-and-run experience: with --compile enabled, the public reference implementation reaches about 16 fps on a single H100. The paper’s 20.7 fps figure comes from an optimized pipeline. The model card lists output dimensions of 576×576 or 768×512. Also note that the paper’s abstract labels the smaller model as 576p, while its main and supplementary results tables say 480p; the specification is inconsistent. Throughput figures alone also do not tell us how long it takes for the first frame to appear. Model card, paper tables
Reproduction resources are available, but data and licensing remain limited
Kyutai’s official account lists two AURA models and a dataset, and the smaller model card provides weight information, a code entry point, and run commands. However, its weights are licensed under CC BY-NC 4.0, which limits use to non-commercial purposes. Developers looking to integrate it into a commercial assistant cannot adopt it based on speed alone. Official resource list, model card and license
The released dataset is also narrower in scope than the paper describes. The dataset card explicitly says that only part of it is currently public: the available material consists of generated intermediate states from the small model’s teacher, with 30,000 training samples and 200 validation samples, but no decoded videos. The large-model teacher videos, audio tracks, and intermediate states described in the paper are still to be released. Training audio, reference images, and prompts must also be obtained separately from Hallo3. External researchers therefore have a starting point for tracing the work, but the current dataset should not be treated as the paper’s complete materials. Dataset card