VoxCPM2 technical report released: 2 million hours of training audio, one model for multiple application scenarios
Since ModelBest, together with Tsinghua University and OpenBMB, released and open-sourced VoxCPM2 in April this year, VoxCPM2 has quickly attracted widespread attention at home and abroad as a new force among domestic speech large models.
To date, VoxCPM2 has surpassed 400,000 total open-source downloads, placing it among the top performers in the small speech model category. It is estimated to save each developer $264 per year (the ElevenLabs annual fee, nearly RMB 1,800)!
Recently, the VoxCPM2 technical report was released, disclosing the underlying technical ideas behind its four major features (30 languages + 9 Chinese dialects, voice design, general controllable voice, and film-grade high audio quality).
➤ VoxCPM2 technical report
🔗 https://arxiv.org/pdf/2606.06928
As a fully open-source, multilingual, controllably generative speech foundation model with only 2B parameters, VoxCPM2 relies on its innovative hierarchical continuous latent modeling paradigm to bypass the traditional reliance on "discrete speech tokenizers." With up to 48kHz ultra-high audio fidelity, it gives developers and users in the AI speech field a more intuitive voice generation experience.
Three major technical breakthroughs
In the technical report, VoxCPM2 presents the three key technical breakthroughs behind its strong performance.
01 A different path: breaking the "discrete" limit
Traditional TTS speech systems built on large language models mostly convert speech into "discrete tokens" like text (discrete speech tokenizers). Although this approach can directly reuse the predictive power of large models, the quantization process erases acoustic details such as breathing, vibrato, and emotional ups and downs.
To compensate, many speech models adopt a complex "two-stage" architecture: first let the large model predict coarse tokens, then use another diffusion model to "imagine" the acoustic details. It is like drawing a sketch and handing it to someone else to color in — the process is cumbersome and prone to "inconsistent style," producing electrical artifacts and noise.
VoxCPM2 instead chose a harder but more elegant technical route: Hierarchical Continuous-Latent Modeling.
VoxCPM2 uses no external discrete speech tokenizer. It trains end-to-end directly in a continuous acoustic latent space. This means that during transmission and processing, tiny tremors and gradual emotional shifts are preserved in full with the fineness of an "analog signal," restoring the emotional color behind the voice as much as possible.
To balance computational efficiency and ultimate audio quality, VoxCPM2 introduces an asymmetric audio autoencoder (AudioVAE V2):
Efficient encoding: At the input end (Encoder), it operates at a 16kHz sampling rate, compressing redundant audio information so the large model's inference sequence is extremely compact (an autoregressive token rate of only 6.25 Hz).
Extreme reconstruction: At the output end (Decoder), a deeper causal CNN stack and wider channels directly reconstruct 48kHz ultra-high-fidelity audio.
In plain terms, this is like "transmitting in an ultra-lightweight 1080P format while projecting losslessly on a 4K giant screen."
This implicit super-resolution design keeps large-model inference smooth and efficient while giving the final voice unmatched clarity and presence.
In addition, compared with the first-generation VoxCPM, VoxCPM2 broadly widens the model's internal "pipeline":
RALM fusion upgrade: FSQ semantic states and local encoder embeddings move from simple "element-wise addition" to a learnable concatenation-projection, letting the model discover the best point of semantic-acoustic fusion on its own.
LocDiT prefix optimization: The diffusion module (LocDiT) receives higher-bandwidth conditioning signals, effectively avoiding early information collapse.
No positional encoding in RALM (NoPE): This subtle change significantly reduces the risk of overfitting during long-text generation and greatly improves the stability of long-audio generation.
02 Three-stage training on 2 million hours of audio data
VoxCPM2 was trained on 2 million hours of audio.
What does 2 million hours mean? It is equivalent to a human speaking nonstop, without eating or drinking, for 228 years.
To avoid losing direction in this flood of data, the team designed an elaborate three-stage progressive curriculum:
Stage 1: Multilingual pre-training
Quickly establishing basic pronunciation and intonation across 30 languages and 9 major Chinese dialects.
Stage 2: Joint controllable pre-training
Introducing control data with natural-language descriptions to train voice design and reference cloning. To prevent text content from being "hard-bound" to intonation style (for example, cheerful words always mapping to a cheerful voice), the team also used the model itself to generate content-decoupled clone samples for reverse training, greatly improving the purity of style control.
Stage 3: High-quality annealing fine-tuning (SFT)
Using the cleanest, most emotionally rich curated dataset for final polish and rebirth.
03 One model for every scenario
Most speech models need different small modules or fine-tuned models for tasks such as dialect cloning, voice design, and continuation cloning. VoxCPM2 breaks this technical inertia with strong architectural generality.
VoxCPM2 introduces a "unified sequence organization" design. Whether it is basic text-to-speech, voice design, style-controllable cloning, or high-fidelity continuation, the underlying layer uses the same set of parameters, the same objective function, and the same inference pipeline.
The only difference lies in how the input sequence is "arranged" (as shown in Table 2 of the technical report):
Basic TTS: text ➡️ target audio
Voice design: voice description text ➡️ target audio. For example, given "a cheerful young male voice saying…," the model creates an entirely new voice matching that description.
Reference cloning: reference audio/text ➡️ target audio — give it a reference voice to clone the timbre, and use text to direct it to generate the target audio according to the user's intent (such as "speak in an extremely angry tone").
Controllable cloning: reference audio/style description text ➡️ target audio
Continuation cloning: prompt text + target text/prompt audio ➡️ target audio
This design eliminates architectural fragmentation, making VoxCPM2 a "Swiss Army knife" of voice that can create any voice the user imagines.
Strong performance across multiple tasks
On the zero-shot voice cloning benchmark Seed-TTS-Eval, VoxCPM2 shows dominance on par with closed-source commercial models:
On the English test set, WER (word error rate) is as low as 1.84%, and speaker similarity (SIM) reaches 75.3%. On the Chinese test set, CER (character error rate) is an excellent 0.97%, with similarity at 79.5%; even on the notoriously difficult Chinese Hard test set, it outperforms the vast majority of open-source and closed-source solutions.
On the instruction-following benchmark InstructTTSEval (covering acoustic parameter control, descriptive style instructions, and role-play), VoxCPM2 scores 84.2 / 83.2 / 71.4 on the English subsets, surpassing GPT-4o-mini-TTS and comparable open-source models, demonstrating its ability to understand plain language and adjust its voice accordingly:
For multilingual and cross-lingual synthesis, uneven data distribution has long been an industry pain point. VoxCPM2 supports not only 30 major world languages, including English, Japanese, Korean, and French, but also 9 Chinese dialects, including Cantonese, Sichuanese, Shanghainese, and Hokkien.
On the multilingual benchmark MiniMax-MLS-Test, VoxCPM2 ranks first in 22 of 24 languages on "voice similarity," beating well-known speech models such as MiniMax and Qwen3-TTS, showing remarkable cross-lingual voice characteristic retention:
Even for rare languages never seen in training (such as Czech and Romanian), it shows emergent zero-shot transfer capability:
Excellent inference efficiency: runs on a single RTX 4090
Thanks to the extremely low autoregressive token rate (6.25 Hz), VoxCPM2 is also very hardware-friendly:
Ultra-low latency and VRAM usage: On a single consumer GPU, an NVIDIA RTX 4090 (24 GB), the standard PyTorch implementation achieves an RTF (real-time factor) of 0.30, with peak VRAM usage of only ~8 GB.
Extreme acceleration: With the specially optimized Nano-vLLM inference engine, RTF drops to an astonishing 0.13. That means generating 10 seconds of high-fidelity speech takes only 1.3 seconds.
Native streaming output: The causal structure of TSLM and RALM and the local-block design of LocDiT naturally suit low-latency "generate while playing" interaction, making it a strong speech foundation for applications such as intelligent customer service and real-time translation.
In the technical report, VoxCPM2 not only discloses all technical details but also releases the model weights, fine-tuning code, and inference tools under the Apache 2.0 license.
Voice is an extension of human emotion. With the open-source release of VoxCPM2, we hope that in the near future, the warmth of human-machine interaction will no longer be determined by cold numbers, but carried in natural, warm, delicate whispers with breath.
👉 Project and open-source links:
➤ Technical report
🔗 https://arxiv.org/pdf/2606.06928
➤ GitHub
🔗 https://github.com/OpenBMB/VoxCPM/
➤ Hugging Face
🔗 https://huggingface.openbmb.com/model/openbmb/VoxCPM2
➤ Try it online
🔗 https://voxcpm.modelbest.cn/