A first in China: ModelBest open-sources ten-million-scale SFT data and the largest Chinese synthetic dataset, revealing the core data behind MiniCPM5-1B
From May 25 to 29, ModelBest and OpenBMB jointly held "On-Device Large Model Open-Source Week," unlocking one killer capability of on-device large models each day. The peak of on-device large models is not just the tip of the iceberg, but the whole iceberg. Today is the fifth installment: new datasets in the UltraData series.
On May 29, ModelBest, together with Tsinghua University and the OpenBMB open-source community, released and open-sourced two new datasets: Ultra-FineWeb-L3 and UltraData-SFT-2605.
Ultra-FineWeb-L3 contains high-quality Chinese + English synthetic web data, exceeding 600B tokens in total, with Chinese accounting for 200B+ tokens. It is currently the largest open-source Chinese pre-training synthetic dataset; UltraData-SFT-2605 is China's first open-source ten-million-scale SFT dataset that includes both deep-thinking and non-thinking annotations.
The training process of MiniCPM5-1B is a complete practice run of the UltraData tiered governance system. Both new datasets, Ultra-FineWeb-L3 and UltraData-SFT-2605, have been fully validated in the MiniCPM5-1B training pipeline, covering the entire chain from pre-training annealing to post-training SFT.
Both datasets are built on ModelBest's original UltraData tiered data governance system, providing the large-model industry with autonomous, controllable, high-quality, and reproducible data infrastructure. They are now fully available on the UltraData website and Hugging Face, free for developers worldwide.
UltraData tiered governance: from "stacking scale" to "refining data"
Training data for large models is like crude oil: without refining, it cannot directly drive an engine. As model architectures converge and high-quality public data resources approach exhaustion, relying solely on scaling data volume is no longer sustainable. Different training stages have markedly different requirements for data quality, quantity, and distribution, calling for targeted data strategies.
In February this year, the L0-L4 five-level tiered data governance system proposed by ModelBest, Tsinghua University, and OpenBMB was designed to solve exactly this problem.
From raw data (L0) through basic filtering (L1), model-based refinement (L2), synthesis and augmentation (L3), to data finally ready for orchestration (L4), each level corresponds to clear processing standards and application scenarios, avoiding one-size-fits-all coarse processing. In systematic experiments across English web, Chinese web, math, and code, model performance improved continuously as data quality rose from L1 to L3, and tiered training improved average performance by 1.49 percentage points over mixed training.
Without systematic data governance, there is no reproducible AGI evolution. The open-sourcing of the L3 dataset is a key step in moving this methodology from theory to practice and extending it from the math vertical to general web data and post-training SFT.
Ultra-FineWeb-L3: making web text go from "readable" to "learnable"
Ultra-FineWeb-L3 is a synthetic dataset that systematically applies L3 refinement techniques to general web data, with over 600B tokens, including 400B+ English tokens and 200B+ Chinese tokens. It is also currently the world's largest Chinese pre-training synthetic dataset, resolving the long-standing industry pain points of scarce and unevenly distributed high-quality Chinese pre-training data.
Using the L2-refined high-quality web dataset Ultra-FineWeb as seed data, Ultra-FineWeb-L3 uses multiple models for QA pair generation and multi-style rewriting, upgrading massive web text into structured, information-dense pre-training data better suited to model learning. Its core goal is not to increase data volume, but to improve the learnability of each unit of data.
At the same training budget, Ultra-FineWeb-L3 significantly outperforms other high-quality datasets such as FineWeb-edu and FinePhrase on English tasks including ARC-E, HellaSwag, and OpenbookQA, as well as Chinese tasks such as CMMLU, and its lead continues to widen in the later stages of training.
As the core training data for MiniCPM5-1B's annealing stage, it provides high-density fuel for the model's "last mile" capability leap in late training.
If the previously released UltraData-Math was the best practice of the UltraData tiered governance system in the math domain, then Ultra-FineWeb-L3 is a further extension of this methodology to general web data.
Its significance is not just providing a larger-scale synthetic web dataset, but turning "readable" web text into "learnable" training data.
UltraData-SFT-2605: fully open-sourced ten-million-scale deep-thinking and non-thinking SFT
If pre-training determines the breadth of a model's knowledge, the SFT stage determines whether the model can follow instructions precisely and reason step by step to answers for complex problems.
UltraData-SFT-2605 is China's first open-source full SFT dataset at the ten-million scale with complete annotations for both deep thinking and non-thinking, covering math, code, knowledge, instruction following, and other domains.
The dataset builds two types of data in every domain and at every difficulty level:
Non-thinking data: corresponds to direct answering in fast-response scenarios
Deep-thinking data: corresponds to reasoning, planning, and verification in complex task scenarios, with complete chain-of-thought annotations
Through this dual-capability parallel design, the model receives targeted training signals for different usage scenarios.
Transparent end-to-end quality governance is another highlight of UltraData-SFT-2605.
Even within L3 data, samples still differ in question value, answer quality, reasoning clarity, and benchmark contamination risk, so clear quality screening standards are also needed within L3.
To this end, UltraData-SFT-2605 introduces a complete data governance process during construction: from query construction and screening, to answer quality control, to benchmark decontamination and training validation, ensuring as much as possible that the data entering final training is clean and genuinely effective.
Diagram of the SFT data construction and validation pipeline
Developers can trace the processing of every data item and understand the basis for retaining, augmenting, and validating it, providing a reproducible data foundation for 1B-class on-device large models to acquire strong reasoning capabilities. On this basis, developers can quickly train on-device large models with deep thinking capabilities without building a complex reasoning annotation system from scratch.
More than scale: a reusable data governance methodology
The development of large models is entering a new stage, and data science must move from extensive scale expansion to refined data governance and utilization. The UltraData tiered governance system provides exactly this systematic approach to the transition.
UltraData has already accumulated a rich foundation in data governance and open source. Its open community website systematically organizes prior work on high-quality datasets such as UltraChat and UltraFeedback, and has open-sourced 2.4T tokens of data from its experiments along with four data governance tools.
Among them, UltraData-Math built a tiered math data system from L1 web corpus to L3 reasoning chains, with a total scale of 290B+ tokens, including 88B tokens of L3 synthetic math data. It is currently the largest open-source math pre-training dataset and topped the Hugging Face Trending chart at No. 1.
Ultra-FineWeb topped the Hugging Face Trending chart for two consecutive weeks, with cumulative downloads exceeding 500,000.
The newly open-sourced Ultra-FineWeb-L3 and UltraData-SFT-2605 are landmark results of extending this tiered governance methodology from the math vertical to general web data and the post-training SFT stage.
Open source is not the finish line, but the starting point for a long-term data ecosystem
High-quality data is becoming a key variable in bringing on-device intelligence to reality. For on-device manufacturers, UltraData's open source means they no longer need to repeatedly invest huge costs in web synthesis and SFT data cleaning; they can directly reproduce MiniCPM5-1B-class capabilities.
High-quality L3 data brings 1B models close to larger models on math, code, reasoning, and other tasks, accelerating the deployment of on-device intelligence in phones, PCs, cars, and other scenarios. High-quality data means fewer training tokens to achieve the same result, indirectly saving compute and memory — a key link in moving on-device large models from technical validation to large-scale deployment.
ModelBest has always upheld the philosophy of "open source," providing community developers with end-to-end support from data acquisition to model training, deeply exploring how to efficiently quantify data value and precisely match the training needs of each model stage, further building "data <-> model co-evolution" and forming a virtuous loop in which models feed back into data governance and high-quality data further optimizes models.
The launch and open-sourcing of the UltraData series is a new starting point for continuous iteration.
In the future, UltraData will gradually open more pre-training L1 / L2 / L3 data, stronger general SFT data, Agent-SFT data, and RL data. The related governance algorithms and models will also continue to be open-sourced, building a data ecosystem that moves from "one-time release" to "continuous iteration," so high-performance AI can reach everyone faster and more naturally.
➤ UltraData website
🔗 https://ultradata.openbmb.cn
➤ Hugging Face
🔗 https://huggingface.co/collections/openbmb/ultradata
➤ Ultra-FineWeb-L3
🔗 https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3
➤ UltraData-SFT-2605
🔗 https://huggingface.co/datasets/openbmb/UltraData-SFT-2605
➤ ForgeTrain
🔗 https://github.com/OpenBMB/ForgeTrain