Governance system + new data release! The UltraData tiered data management system launches to empower AGI with scientific governance
The history of artificial intelligence is essentially a history of the evolution of data-driven strategies and data utilization. Each paradigm shift both extends and reconstructs the previous stage's data-driven strategy and evolves new ways of using data, driving leaps and emergent capabilities in models.
Illustration of the evolution of data-driven strategies and data utilization
AGI development has gone through four stages: symbolic learning, supervised learning, unsupervised learning, and feedback learning. Looking back, the current mainstream paradigm is "Data-Driven Learning," in which expanding data scale unidirectionally drives model capability. As models grow stronger, we believe AI development is moving toward a new stage: "Data-Model Co-Evolution," forming a virtuous loop in which models feed back into data governance and high-quality data further improves models.
To adapt to this paradigm shift, THUNLP, ModelBest, and OpenBMB jointly officially release the model-driven L0-L4 Tiered Data Management system, aiming to provide systematic data science support for the path to AGI. To validate the effectiveness of tiered data governance, we conducted systematic experiments in four domains — English web pages, Chinese web pages, mathematics, and code — and observed that model performance improves consistently as data quality rises from L1 to L3.
During the experiments, we systematically built UltraData-Math, a large-scale, high-quality pre-training dataset for mathematics. It contains the largest open-source high-quality L3 synthetic math data available today, outperforms other open-source data on evaluation benchmarks, and provides a new frontier governance paradigm for vertical-domain math data.
To further build a data governance community ecosystem, we launched the UltraData open community website. The site organizes earlier high-quality data work such as UltraChat and UltraFeedback, and open-sources 2.4T tokens of data and 4 data governance tools from the experiments.
Open-source data and tools
➤ Related links
Tiered data management paper:
🔗 https://ultradata.openbmb.cn/blog/position-paper
UltraData website:
🔗 https://ultradata.openbmb.cn
Hugging Face page:
🔗 https://huggingface.co/collections/openbmb/ultradata
UltraData-Math:
🔗 https://huggingface.co/datasets/openbmb/UltraData-Math
The UltraData tiered data management system
Existing data governance methods tend to focus on specific data processing tasks (such as filtering, selection, and editing), but these methods cannot meet the systemic requirements of training a large language model across its full lifecycle. There are three main motivations for building a systematic data governance system based on model training needs:
High-quality public data resources are approaching exhaustion; future model evolution can no longer rely solely on growing data scale. Data science must move from extensive scale expansion toward refined data governance and utilization to unlock the deeper value of existing data.
Different training stages (such as pre-training knowledge injection and fine-tuning instruction alignment) have significantly different requirements for data quality, quantity, and distribution, making it urgent to build different training data for each stage's learning needs.
Data governance must strike an optimal dynamic balance between cost and model benefit. In the early stages, lightweight, low-cost governance methods (such as heuristic filtering) should be used, while deeper governance stages should use more refined, higher-cost methods (such as LLM labeling). Strategically allocating high-value data to key points such as mid-training or learning-rate decay can maximize the marginal effect of data while controlling total cost.
Based on these motivations, we built a five-level L0-L4 data tiering system to make data processing more targeted. From raw data (L0) through basic filtering (L1), model-based selection (L2), and synthesis and augmentation (L3), to data that can finally be used directly for organization (L4), each level corresponds to clear processing standards and application scenarios, avoiding one-size-fits-all extensive processing and maximizing the training value of each unit of data.
The core definitions, processing methods, and application scenarios of each level are as follows:
L0 - Raw Data: Initial data (PB scale) without substantive processing, containing large amounts of noise and duplication. After collection and parsing, it serves as the underlying reserve (examples: Common Crawl web pages, PDF documents) and is not used directly for training.
L1 - Filtered Data: Basic cleaned data after heuristic rule filtering and deduplication, with standardized format but uneven semantic quality (examples: FineWeb, DCAD); provides the foundation for subsequent processing.
L2 - Selected Data: High-information-density data selected through model scoring and labeling, with a clear domain and coherent logic (examples: Ultra-FineWeb, FineWeb-edu); contributes significantly to model capability improvement.
L3 - Refined Data: Structured data produced through rewriting, synthesis, and human annotation, with clear semantics and excellent quality (examples: Ultra-Chat, UltraFeedback); suited for advanced training such as mid-training, SFT, and RL.
L4 - Organized Data: Ordered data after curation, trust verification, and unified organization, with clear structure and retrievability (examples: Wikidata, UltraData-Arxiv); directly supports RAG applications.
Illustration of the tiered governance system
01 Validating the tiered system
We conducted extensive experiments to demonstrate the effectiveness of the tiered system. We selected representative L1 to L3 data from four domains — English web, Chinese web, math, and code — and used a rapid validation strategy on MiniCPM-1.2B, sampling 10B tokens from each single data source to validate data quality.
L1-L3 experimental data selection (* marks open-source data)
The experimental results are shown in the table below. Across domains, model performance shows a clear and consistent upward trend from L1 to L3, validating that the proposed tiered data governance system effectively captures meaningful data quality tiers. The strict L3 > L2 > L1 performance ordering holds consistently in all experiments. These results confirm the system's effectiveness in building high-quality data: even under a budget-constrained validation setup, it still produces clear and stable performance improvement signals.
Training results with L1-L3 data in English web, Chinese web, math, and code domains
02 Multi-stage model training based on the tiered system: To validate the dynamic impact of the tiered system on model training, we designed a comparative experiment evaluating mix training versus tiered training. We mixed data from four domains at a fixed ratio of 50% English web, 25% Chinese web, 7% math, and 18% code, and trained from scratch on MiniCPM-1.2B for 120B tokens. The mix training strategy uses a single-stage setup, mixing L1, L2, and L3 at a 1:1:1 ratio into a unified data pool. The tiered training strategy divides training into three consecutive 40B-token stages, using L1, L2, and L3 in sequence.
Comparison of mix training and tiered training results
The table above gives a detailed comparison of the two strategies across evaluation benchmarks. Compared with mix training, tiered training improves overall average performance by 1.49 percentage points and achieves significant gains in all four main evaluation domains: English, Chinese, math, and code.
Average scores of mix training and tiered training at each training checkpoint
The figure above further reveals the advantages of tiered data governance from the dynamic perspective of the training process. In the early stage, both strategies show similar growth, rising from about 24.7pp to about 28.3pp; in this stage, tiered training mainly uses L1 data. In the later stage, tiered training gradually introduces high-quality data selected by the L2 model and optimized through L3 synthesis, and the performance curve shows a continuous, steady rise from 28.35pp to 31.66pp, a gain of 3.31pp. In contrast, mix training slows markedly, rising only from 28.26pp to 30.17pp, a gain of about 1.91pp.
The results show that organizing 120B tokens of training data by L1, L2, and L3 more effectively improves comprehensive model performance across knowledge understanding, logical reasoning, and domain capability, providing solid experimental support for tiered data management.
UltraData-Math: high-quality math pre-training data
Based on the UltraData tiered data management system, we conducted in-depth data governance and rigorous experimental analysis on math data, building the largest L3 synthetic math dataset to date. The experimental results demonstrate the effectiveness of our approach. We have open-sourced the full dataset and tiered governance tools in the OpenBMB, Hugging Face, and other communities.
Specifically, we mainly used the following governance methods:
L0: We developed the math parser UltraData-Math-Parser based on magic-html, combining w3m layout-preserving rendering with a multi-level fallback strategy to standardize formats such as MathML, KaTeX, and AsciiMath into LaTeX.
L1: We cleaned noisy data with heuristic rules and performed document-level deduplication.
L2: We used a proprietary large model to label seed data and distilled it into a lightweight embedding classifier, enabling efficient quality tiering across the entire corpus.
L3: We built UltraData-Math-Generator to produce structured content through various forms of data rewriting, synthesis, and refinement (such as Q&A, multi-party dialogue, multi-style rewriting, and knowledge-backed textbooks), ensuring clear reasoning and learnable knowledge.
Parsing results from UltraData-Math-Parser
After full-pipeline cleaning and reconstruction, the data shifted from "broad coverage" to "deep focus": we ultimately obtained 290B of tiered math data. UltraData-Math-L1 contains 170.5B tokens of web math corpora; UltraData-Math-L2, high-quality math data filtered by a quality model, contains 33.7B tokens; UltraData-Math-L3, high-quality synthetic math data, contains 88B tokens in multiple formats (such as Q&A, multi-turn dialogue, and knowledge textbooks).
💡 Experimental validation and results
To validate data quality, we designed a series of controlled experiments based on MiniCPM-1.2B. All experimental models were trained on 100B tokens.
Experimental data shows that using the UltraData-Math parser at L0 produced a marked divergence in downstream task performance: our parsing approach reached 28.72 on the MATH dataset, outperforming trafilatura (28.08pp) and magic-html (26.58pp). This directly demonstrates that the completeness of parsed data is crucial for models to understand complex mathematical logic.
Comparison of L0 parser experiment results
We compared our final data, UltraData-Math-L3, against state-of-the-art open-source math datasets (Nemotron-CC, MegaMath, FineMath) through independent training. Results show that UltraData-Math sets new performance records for same-scale models on multiple core benchmarks. Thanks to L0-L3 tiered governance, the model not only leads Nemotron-CC 4plus by a wide margin on the MATH benchmark (+3.62 points), but also demonstrates outstanding reasoning performance on multidimensional math evaluations such as GSM8K, Math-Bench, and R-Bench-T, strongly validating the significant gain of L3 synthetic data for complex mathematical chain-of-thought (CoT). Meanwhile, the model shows significant gains on the MBPP code generation task (49.27pp) while maintaining robust general knowledge (MMLU), achieving a dual breakthrough in mathematical depth and logical breadth.
Comparison of math L3 data experiment results
The UltraData open community website
We are proud to launch the UltraData open community and official website. We will promptly publish newly released governance tools, datasets, and research updates on the site.
01 Full-pipeline governance tools
The platform aggregates data governance tools for every tier: from UltraData-Parser at L0 (supporting parsing of heterogeneous content such as web pages and PDFs), to UltraData-Cleaner for L1 filtered data (dynamically composing operators for heuristic rule filtering and deduplication), to UltraData-Selector for L2 selected data (screening high-value samples through model labeling), to UltraData-Generator for L3 synthesized and augmented data (enabling diverse data rewriting and synthesis), and finally UltraData-Organizer for L4 organized data (building and validating structured knowledge bases). Each tool corresponds to clear governance methods and data types, covering the full pipeline from raw data to ordered data after unified organization and standardized validation.
Online demo of UltraData-Math-L0-Parser
We have also deployed these governance tools on platforms such as Hugging Face Spaces, so developers can quickly experience them without deploying anything themselves.
02 Datasets across every tier
The site brings together past achievements and the latest practices in data governance from the OpenBMB community, such as DCAD-2000, L1 multilingual foundational data; Ultra-FineWeb, L2 core pre-training data; UltraChat and UltraFeedback, L3 data dedicated to SFT/RL; and the UltraData-Math dataset series. Every included dataset has been validated by the open-source community and widely praised.
DCAD-2000: 222k cumulative downloads, covering 2,282 languages and 159 writing systems
Ultra-FineWeb: 406k cumulative downloads, No. 1 on Hugging Face Datasets Trending for 2 consecutive weeks
UltraChat: used by 500+ models worldwide according to Hugging Face statistics, ranking 7th
UltraFeedback: used by 1,000+ models worldwide according to Hugging Face statistics, ranking 4th
Conclusion
From the paper proposing a scientific method, to the platform delivering tools, to the datasets providing usable resources, we have built a complete loop from "theory → practice → application." We know well that the future of data science lies not in isolated accumulation but in open collaboration.
In the future, we will unite more academic and industry partners. We sincerely invite developers worldwide to grow the UltraData ecosystem together — share UltraData's high-quality data, co-build a scientific governance system, and co-create new possibilities for AGI practice, looking forward together to high-quality data creating greater value through more open collaboration.