VoxCPM2

VoxCPM Speech Series

Use your own speech data to explore speaker, style and domain adaptation. Start with the data manifest and audio checks, choose LoRA or full-parameter fine-tuning, then use text that was not part of training to validate timbre, intelligibility and generalization.

VoxCPM2 supports both LoRA and full-parameter fine-tuning, which use the same training script and JSONL data format. Most personalized speaker, style or domain tasks should start with LoRA; evaluate full-parameter fine-tuning only when LoRA lacks capacity and you have sufficient data and compute.

Choose a training method

Goal	Data size	Recommended method
Single-speaker cloning	5–50 high-quality clips; per the official docs, 5–10 minutes of audio is enough to get started	LoRA, starting with r=32
Domain or style adaptation	50–500 clips	LoRA; you can evaluate r=32–64
Adding a new language	The official docs recommend 500+ hours	Full-parameter fine-tuning, mixing in some Chinese and English data to mitigate forgetting
Large-scale customization where LoRA lacks capacity	Ample and diverse data	Full-parameter fine-tuning

The reference VRAM given in the official fine-tuning docs is about 20 GB for VoxCPM2 LoRA and about 40 GB for full-parameter training. These are approximate values for the example configurations; batch size, clip length, gradient accumulation and the device will change actual usage.

Prepare the repository and model

Bash

git clone https://github.com/OpenBMB/VoxCPM.git
cd VoxCPM
pip install -e .

python -c "from huggingface_hub import snapshot_download; snapshot_download('openbmb/VoxCPM2', local_dir='./pretrained_models/VoxCPM2')"

pretrained_path in the training config points to a local model directory, not a Hub model ID used only for inference.

Prepare JSONL data

The training manifest has one JSON object per line. audio and text are required; ref_audio, duration and dataset_id are optional.

JSON

{"audio":"data/audio_001.wav","text":"第一条音频的逐字转写。","duration":4.2}
{"audio":"data/audio_002.wav","text":"第二条音频的逐字转写。","duration":6.8,"dataset_id":0}

Check	Requirement
Clips	The official docs recommend 3–30 seconds; clips under 1 second tend to be unstable, while overly long clips increase VRAM usage and may be filtered out by max_batch_tokens.
Transcripts	Must match the audio word for word; typos, missing words and extra filler words all hurt cloning quality and text adherence.
Audio	WAV recommended; single speaker, low noise, low reverberation, consistent volume. The loader resamples automatically.
Training sample rate	The VoxCPM2 config uses sample_rate: 16000, matching the AudioVAE encoder input; out_sample_rate: 48000 is the decoder output.
Validation set	Hold out speech content that does not enter the training set, for overfitting and generalization checks.
Authorization	Record speaker consent, data source, license scope and a deletion mechanism.

Reference-audio conditioned sample:

JSON

{"audio":"data/target.wav","text":"目标音频转写。","ref_audio":"data/same_speaker_reference.wav"}

ref_audio should be another clean recording from the same speaker with the same timbre as the target audio, used as the speaker condition for voice cloning. It does not need to be an unseen sample from the validation set, but it should not point to audio from a different speaker or with a noticeably different timbre.

The official fine-tuning docs recommend that 30%–50% of training samples include ref_audio and the rest omit that field, so as to preserve both zero-shot generation and reference-audio-conditioned voice cloning. This ratio is an official recommendation; you should still confirm with an independent validation set whether it suits your specific data size and task.

Bash

voxcpm validate   --manifest train.jsonl   --sample-rate 16000

LoRA fine-tuning

Copy the official config to create an experiment config, then change the model, data and output paths; do not overwrite the repository examples directly.

Bash

cp conf/voxcpm_v2/voxcpm_finetune_lora.yaml conf/my_voxcpm2_lora.yaml

YAML

pretrained_path: ./pretrained_models/VoxCPM2
train_manifest: ./train.jsonl
val_manifest: ./val.jsonl
sample_rate: 16000
out_sample_rate: 48000
batch_size: 2
grad_accum_steps: 8
learning_rate: 0.0001
max_steps: 1000
save_interval: 500
save_path: ./checkpoints/voxcpm2_lora
tensorboard: ./logs/voxcpm2_lora
lora:
  enable_lm: true
  enable_dit: true
  enable_proj: false
  r: 32
  alpha: 32
  dropout: 0.0

Bash

# 单卡
python scripts/train_voxcpm_finetune.py   --config_path conf/my_voxcpm2_lora.yaml

# 多卡
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4   scripts/train_voxcpm_finetune.py   --config_path conf/my_voxcpm2_lora.yaml

enable_dit is critical to audio quality; keep enable_proj off in most scenarios. The official docs recommend starting speaker cloning at r=32, and evaluating r=64 for style or language adaptation.

Full-parameter fine-tuning

Bash

cp conf/voxcpm_v2/voxcpm_finetune_all.yaml conf/my_voxcpm2_full.yaml
python scripts/train_voxcpm_finetune.py   --config_path conf/my_voxcpm2_full.yaml

The full-parameter config has no lora section, and its example learning rate is 1e-5, about 1/10 of the LoRA example. Full-parameter training is more prone to catastrophic forgetting and overfitting; when adding a new language, the official docs recommend mixing in 10%–20% Chinese and English data.

Monitor training

Signal	Action
loss/diff and loss/stop	Watch the training and validation trends together; when validation loss starts rising again, prefer an earlier checkpoint.
Generation doesn't stop	Check loss/stop, sample durations, text alignment and data anomalies.
Timbre matches but the text is ignored	A typical sign of overfitting; reduce the steps or learning rate, and increase text and prosody diversity.
Out of VRAM	Lower batch_size or max_batch_tokens and increase gradient accumulation; prefer LoRA.

Validate checkpoints

Bash

# LoRA checkpoint
python scripts/test_voxcpm_lora_infer.py   --lora_ckpt checkpoints/voxcpm2_lora/step_0000500   --text "这是一条未出现在训练集中的验证文本。"   --output lora_eval.wav

# 全参数 checkpoint
python scripts/test_voxcpm_ft_infer.py   --ckpt_dir checkpoints/voxcpm2_full/step_0000500   --text "这是一条未出现在训练集中的验证文本。"   --output full_eval.wav

At a minimum, compare timbre similarity, text intelligibility, generalization to unseen text, stability on long and short sentences, regression of base-model capabilities, and variation across multiple generations. Don't just listen to sentences from the training set.

LoRA WebUI and hot-swapping

Bash

python lora_ft_webui.py

The WebUI is suited to low-barrier experiments. For production training, you should still save the YAML, data revision and code commit. At runtime, you can use load_lora, set_lora_enabled and unload_lora to load, temporarily disable or switch LoRAs without restarting the base model.

Related Resources

Topic	Link
Model overview	VoxCPM2 overview
Official fine-tuning documentation	ReadTheDocs Finetuning
Code repository	OpenBMB/VoxCPM