開源語音模型
Tencent’s AuK Unifies Speech Generation, Content Rewriting, and Acoustic Editing Through a Single Command Interface
AuK brings zero-shot TTS, utterance replacement, voice and emotion adjustment, denoising, and source separation into a single 1.5B-parameter generative model. The distilled AuK-Flash requires only four sampling steps, though its claimed 4.5× speedup and quality advantages still rely primarily on the team’s own experiments.
Teams from Tencent Hunyuan, Shanghai Jiao Tong University, and other institutions submitted the [AuK technical report](https://arxiv.org/abs/2609.08936) on September 8 and released the [code and weights](https://github.com/Tencent-Hunyuan/AuK) the following day. Unlike conventional pipelines that separately chain together TTS, speech inpainting, denoising, and source separation models, AuK accepts natural-language instructions and optional reference audio to handle 16 categories of tasks within a unified system. These include zero-shot and description-based TTS, word or phrase insertion and deletion, lyric rewriting, pitch and speaking-rate control, emotion, timbre, and accent adjustment, as well as vocal enhancement and separation.
The architecture consists of three components: a multimodal language model that supplies semantic conditioning; a VAE jointly trained on speech, general audio, and music to represent acoustic content; and a generator that first processes conditioning through a dual-stream MMDiT before passing it to a single-stream DiT, using a rectified-flow objective to synthesize the output. The team says its training data corresponds to approximately 3.03 billion instruction–audio instances and 1.95 million hours of effective supervision. Training begins with generation-only pretraining, followed by joint learning of generation and editing, and concludes with separate stages of human-preference optimization and reward-based reinforcement learning.
Deployment requirements cannot be estimated from the “1.5B” figure alone: the current implementation must also load a separate VAE and Qwen2.5-Omni-3B encoder, with their checkpoints downloaded independently. The base model allows users to adjust the number of sampling steps and classifier-free guidance. Following consistency initialization and task-routing distillation, AuK-Flash uses a fixed four-step process with CFG disabled; under identical conditions, the developers measured a 4.5× reduction in wall-clock time. The repository already provides command-line, Python, Gradio, ComfyUI, and fine-tuning interfaces, making it convenient for engineering teams to evaluate. However, the paper’s state-of-the-art results still need independent replication across different languages, accents, and noise environments. Open-sourcing the model also does not automatically resolve licensing, impersonation, and watermarking concerns associated with voice cloning.