UltraEval-Audio v1.1 Released: One-Click Reproduction and an Isolated Inference Runtime for More Efficient, More Controllable Evaluation
Audio models have become essential productivity tools, and for researchers, improving their performance and quality is now a top priority. Amid this rapid pace of iteration, researchers commonly run into several problems that drag down R&D efficiency. First, the metrics reported in papers and public reports are often hard to reproduce reliably on a local machine: reproduction pipelines depend on scattered scripts, hidden parameters, and convoluted pre- and post-processing, so the results don't match and the process doesn't run. Second, models differ enormously in their runtime requirements — framework versions, CUDA dependencies, mutually exclusive third-party libraries — and evaluating several models in parallel on a single machine easily descends into dependency conflicts and repeated environment rebuilds. Third, beyond general-purpose speech LLMs, specialized audio models such as TTS, ASR, and Codec models are drawing growing attention from both the community and industry, with popular models and approaches emerging quickly — and demand for reproducible evaluation of these models is rising sharply.
To address these pain points, Tsinghua University, OpenBMB, and ModelBest have jointly released UltraEval-Audio v1.1. Building on the original one-click evaluation of audio models, the new version focuses on one-click reproduction for popular audio models, extends support to specialized TTS/ASR/Codec models and task-specific evaluation, and introduces an isolated inference runtime — lowering the engineering barrier to reproduction while making the evaluation pipeline more controllable and portable.
Notably, as the evaluation tool of choice for high-impact audio and omni-modal models such as MiniCPM-o 2.6 and VoxCPM, the open-source UltraEval-Audio v1.1 will help researchers substantially improve the efficiency of audio model R&D.
UltraEval-Audio: The Industry's First Evaluation Framework for Omni-Modal and Speech LLMs
UltraEval-Audio is an evaluation framework for audio models jointly developed by THUNLP, OpenBMB, and ModelBest. It not only establishes a complete evaluation methodology for the audio LLM field but also turns that methodology into an out-of-the-box engineering framework, refining the audio evaluation system and adding systematic definitions of the evaluation suite (see Table 1).
Yet as the open-source community has exploded, one fact must be acknowledged: evaluation presupposes reproduction, and the cost of reproduction is increasingly a wall standing between models and real-world deployment. UltraEval-Audio v1.1 therefore takes a key step from defining standards to empowering engineering. The new version no longer focuses solely on final evaluation results; it is dedicated to closing the last mile between a model's code repository and an evaluation pipeline.
➤ Related Links
UltraEval-Audio v1.1 Evaluation Framework:
🔗 https://github.com/OpenBMB/UltraEval-Audio/tree/main/replication
UltraEval-Audio v1.1 paper, "UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models":
🔗https://github.com/OpenBMB/UltraEval-Audio/blob/main/assets/UltraEval_Audio\_\_A_Unified_Framework_for_Comprehensive_Evaluation_of_Audio_Large_Language_Models.pdf
One-Click Reproduction for Popular Models: Matching Paper Results, Fast
In audio, reproduction is often as much an engineering challenge as algorithmic innovation. Tangled dependencies in official repositories, vaguely documented preprocessing logic, and temperature or sample_rate values buried in the corner of some script can turn reproduction into an exercise in guesswork.
To break this cycle, UltraEval-Audio v1.1 provides deep adaptation and official reproduction for leading community models such as VoxCPM, MiniCPM-o 2.6, CosyVoice3, GLM-TTS, Kimi-Audio, and Qwen3-Omni. We did more than get the code running: we refactored these models' inference pipelines as white boxes:
Standardized pipeline encapsulation: We dissected the scattered preprocessing scripts (such as audio resampling and feature extraction) and post-processing logic in each model's official repository, then wrapped them in standardized form inside the framework's Prompt Management and Model Deployment modules. Whether it is special handling for an audio tokenizer or fine-grained alignment of a chat template, UltraEval-Audio handles it all at the underlying layer.
WYSIWYG reproduction baselines: In the replication/ documentation directory we provide benchmark evaluation reports aligned with official results, along with one-click reproduction scripts (see the GitHub link). Researchers can obtain metrics closely matching those in the published paper by running a single command.
➤ GitHub Link
🔗 https://github.com/OpenBMB/UltraEval-Audio/tree/main/replication
From now on, researchers no longer need to spend hours debugging environments or guessing at hidden parameters: the path from downloading a model to producing authoritative evaluation results is seamless.
Isolated Inference Runtime: Fully Resolving Dependency Conflicts
When evaluating audio LLMs, researchers face a stubborn paradox: making a single Python environment compatible with every model is all but impossible. Established models may depend on CUDA 11.x and an older torchaudio, while newer models require PyTorch 2.x and the latest Flash-Attention. This version exclusivity forces researchers evaluating multiple models in parallel to reinstall drivers or switch Conda environments by hand — an engineering nightmare that severely slows iteration.
To solve this, UltraEval-Audio v1.1 rebuilds the underlying architecture and introduces an isolated inference runtime. As shown in the architecture diagram, this mechanism physically decouples the evaluation control flow from the model compute flow:
Fully automated environment sandbox: The framework no longer forces model inference to run in the main process. Instead, it reads the configuration file to identify and invoke each model's own Python virtual environment automatically. This means you can evaluate two models with completely conflicting dependencies on the same server at the same time; the system assigns each an independent sandbox so they never interfere.
Robust IPC communication: Model loading and inference processes are encapsulated in isolated environments and exchange data with the main evaluation process only through a lightweight, high-throughput IPC (inter-process communication) protocol.
Decoupling benefits: The evaluator stays extremely lightweight and stable. No matter how the underlying models' inference scripts change or how complex their dependencies become, the stability of the evaluation logic above is unaffected.
Fault isolation: Even if one model's inference process crashes because of an environment problem, the entire evaluation job does not stop.
With this isolation mechanism, UltraEval-Audio truly delivers on "configure once, run anywhere," returning evaluation work to what matters about the model itself rather than endless environment debugging.
Broader Evaluation Coverage: Specialized TTS, ASR, and Codec Models
Beyond general-purpose speech LLMs, we have seen growing interest from both the community and industry in specialized models for TTS (text-to-speech), ASR (automatic speech recognition), and Codec (audio codec). In v1.1 we removed the boundary of evaluating only general audio LLMs and extended evaluation down to three specialized domains — TTS, ASR, and audio codec — building full-stack infrastructure for audio evaluation.
TTS: focus on task diversity. For TTS models we integrate established datasets including Seed-TTS-Eval, CV3-Eval, and Long-TTS, covering typical scenarios such as VC voice cloning and long-form speech synthesis, and providing multi-dimensional quantitative baselines for synthesis accuracy, timbre imitation, and acoustic naturalness.
ASR: broad scenario coverage. For ASR models we support more than ten mainstream datasets, including LibriSpeech, Common Voice, AISHELL-1, and WenetSpeech. Coverage spans clean read speech (AISHELL-1) to complex real-world conditions (WenetSpeech), and single-language (LibriSpeech) to multilingual (MLS, FLEURS), ensuring the results offer broadly useful robustness references.
Audio Codec: a three-dimensional evaluation system. As a low-level component of audio foundation models, a codec's reconstruction quality is critical. To address inconsistent evaluation standards, we built a three-dimensional system covering semantics, timbre, and acoustics, giving model optimization a fine-grained diagnostic tool:
Semantics: WER (word error rate) computed with Whisper-large-v3 and Paraformer-zh, verifying that content is preserved;
Timbre: speaker embeddings extracted with WavLM-large and compared by cosine similarity to measure timbre fidelity;
Acoustics: UTMOS (naturalness) combined with DNSMOS (noise robustness/audio quality) to quantify perceived listening quality objectively.
We provide detailed evaluation and reproduction guides to help researchers get started quickly:
👉 Start your evaluation journey:
https://github.com/OpenBMB/UltraEval-Audio
👉 Reproduction guide:
https://github.com/OpenBMB/UltraEval-Audio/tree/main/replication
Which model matters most to you?
We already support models such as VoxCPM, MiniCPM-o 2.6, CosyVoice3, and Qwen3-Omni, but the universe of speech AI extends far beyond them. Have you ever been excited by a paper only to dread reproducing it? Which star model would you most like to see get one-click reproduction in UltraEval-Audio? We will prioritize the model with the most votes — so the truth no longer arrives late.
➤ GitHub Link
🔗 https://github.com/OpenBMB/UltraEval-Audio/issues/41