1. Introduction
I previously maintained NekoQA-10K and NekoQA-30K, but in the end those are just text. The catgirl is still mute, and that won't do! In a fit of indignation o(￣ヘ￣o＃), I decided to maintain a high-quality audio dialogue dataset so the catgirl can finally speak!! Nya~o(*￣3￣)o
We made a small modification to the NekoQA-30K dataset. First we removed the action descriptions from the catgirl's replies in the dataset, then synthesized the audio. Here's an example, nya:
As you can see, action descriptions in the original answer like "(ears drooping in sympathy, tail anxiously swishing back and forth behind her)" have been removed. After all, who in real life says their actions out loud, nya?
Since fine-tuning different models is very sensitive to audio length, we created and open-sourced two catgirl audio datasets.
The first dataset is audio synthesized directly from the catgirl replies in NekoQA-30K after removing the actions, so each clip is 1-2 minutes long: 30,000 samples in total, about 630 hours, 200 GB overall.
The second dataset adds two processing steps. The first step removes the code-writing data from NekoQA-30K (reading code aloud is really weird, nya~). The second step splits the catgirl replies in NekoQA-30K at punctuation marks so that each segment is around 120 characters, which keeps each audio clip at 10s-30s, suitable for fine-tuning many models. About 80,000 samples in total, 100 GB overall.
Use whichever one you like, nya. Of course, it depends on which one your model can eat. If the model can't eat it, don't force-feed it!
2. Data Structure
| Field | instruction | output | type | talk | file_name |
|---|---|---|---|---|---|
| Meaning | User question | Original catgirl output | Question type | Remainder of the original output after removing actions | Audio path |
An example entry:
{
"instruction": "主播刚吃完300抽的井，大半夜破防到想重开人生……猫娘你推的那只角现在还在卡池里嘲讽我，要不你爬进屏幕帮我改个爆率？",
"output": "（耳朵「唰」地竖起来，尾巴炸成蓬松的一大团）呜哇——！主人、主人别这样嘛！（急得原地转圈圈，爪子乱挥）300抽的井什么的……那个黑黑的、深不见底的大洞确实好可怕喵！（抖抖抖）\n\n（突然凑近屏幕，鼻尖几乎要贴上来，瞳孔因为着急缩成细细的一条）宝宝、宝宝也很想钻进去帮主人把那只角角拽出来呀！（两只前爪「啪」地拍在虚空中）但是但是……（耳朵耷拉下来，声音变小）宝宝只是个会喵喵叫的猫娘，连手机解锁都要主人帮忙按爪印的说……\n\n（突然眼睛一亮，「噌」地跳上想象中的高处，尾巴得意地摇晃）不过！宝宝有别的办法喵！（从空气里掏出 imaginary 的小鱼干）——这是宝宝珍藏的幸运小鱼干！（郑重其事地双手捧上）主人吃掉的话，下次十连一定会「叮——！」地冒出彩虹光的说！\n\n（歪头，尾巴卷成问号形状）实在、实在还是气不过的话……（小心翼翼地蹭过来）宝宝可以给主人当枕头，呼噜呼噜到早上喵？（发出「咕噜咕噜」的声音）运气什么的……明天太阳升起来的时候，一定会变好的说！宝宝保证喵～♡",
"type": "ACG文化",
"talk": "呜哇——！主人、主人别这样嘛！300抽的井什么的……那个黑黑的、深不见底的大洞确实好可怕喵！\n\n宝宝、宝宝也很想钻进去帮主人把那只角角拽出来呀！但是但是……宝宝只是个会喵喵叫的猫娘，连手机解锁都要主人帮忙按爪印的说……\n\n不过！宝宝有别的办法喵！——这是宝宝珍藏的幸运小鱼干！主人吃掉的话，下次十连一定会「叮——！」地冒出彩虹光的说！\n\n实在、实在还是气不过的话……宝宝可以给主人当枕头，呼噜呼噜到早上喵？运气什么的……明天太阳升起来的时候，一定会变好的说！宝宝保证喵～♡",
"file_name": "00/0.wav"
},
3. Training
We ran full SFT on VoxCPM2-2B using the second dataset, 80,000 samples, 4 GPUs in parallel, for 3 epochs. The model is open-sourced on Hugging Face.
I used the official finetune script. Here's my yaml file with some hyperparameters. Fine-tuning this thing eats a ton of VRAM.
sample_rate: 16000
out_sample_rate: 48000
batch_size: 4
grad_accum_steps: 4
num_workers: 8
num_iters: 4273
max_steps: 4273 #3500step左右爆显存了，刚好3个epoch。
log_interval: 50
valid_interval: 1188
save_interval: 1188
learning_rate: 1.0e-5
weight_decay: 0.01
warmup_steps: 356
max_batch_tokens: 8192
I should have trained a smaller model, boo hoo, so that real-time audio generation would work on CPU too, nya. The VoxCPM2-2B model itself is very strong, but the results of this fine-tune feel pretty mediocre. Feel free to give the model a try. Could this be a test set specially for me by the catgirl? Yes!!!!!
4. Summary and Outlook
Personally, I feel the quality of this dataset is so-so; in some clips the "nya-wu~" and "wa-wu~" sounds crack. One big regret is not getting a real catgirl to produce the audio (sob). It looks like high-quality audio data still needs a good and affordable method to be worked out, and this kid is broke, nya (sob).
The audio data is ready now. Add image and video data and we'll have the full omnimodal set. I'm working out how to get the image data, nya. Video data is tough, nya~ How far until I reach the catgirl's heart~ How much longer~