ModelBest Ultra Super-Alignment: Empowering 200+ AI models, 1M+ monthly downloads
Quality data creates quality models. To date, over 200 models worldwide have been aligned using the ModelBest Ultra Alignment Series datasets (including UltraFeedback and UltraChat), with monthly downloads exceeding 1 million. How good is the dataset quality? The real combat effectiveness of large models tells the story.
In some classic "small vs. large" model battles, the ModelBest Ultra alignment datasets are frequently present, demonstrating real-world validation across global large models.
For example, powered by the ModelBest Ultra datasets, Zephyr-7B with a smaller scale outperforms LLaMA2-70B-Chat on numerous metrics; the Ultra datasets also help the "small powerhouse" MiniCPM-2B achieve impressive performance comparable to Mistral-7B.
ModelBest Ultra: Leading Alignment Technology
Why is alignment technology so important? It is arguably one of the most critical tasks in the AGI journey. Like placing a "restraining band" on large models, ensuring their goals, values, and interests align with humans and meet designers' expectations, avoiding harmful consequences beyond control.
ModelBest Ultra alignment technology (Ultra Series) integrates a series of world-leading achievements in large model alignment. Open-sourced by ModelBest AI in collaboration with Tsinghua NLP Lab, it currently includes high-quality datasets UltraFeedback, UltraChat, and large language models UltraLM, UltraRM, UltraCM, etc.
The construction of Ultra Series datasets is based on the team's pioneering "Scalable Diversity" principle. The team believes that the continuous scalability of dataset size and the diversity of instructions and responses in the data determine data quality. To achieve dataset scalability, the team eliminates any human intervention throughout the construction process, with the entire dataset automatically built by large models. To improve diversity and data quality, the team has also designed a series of methods.
Core Datasets:
• UltraFeedback is the first large-scale general AI feedback dataset, pioneering the validation of preference alignment on open-source large models. This large-scale, diverse, fine-grained preference dataset includes 250,000 conversation data points with corresponding preference annotations, where each preference annotation contains fine-grained scores across four dimensions with detailed textual explanations. Among non-community-annotated preference datasets, UltraFeedback ranks first in data scale.
• UltraChat is a high-quality dialogue dataset containing over 1.5 million multi-turn instruction data. Multiple ChatGPT APIs are invoked to converse with each other, generating multi-turn dialogue data.
Technical Innovations:
To validate dataset quality, the team trained a reward model and a critique model based on UltraFeedback to further assist model evaluation and feedback learning:
• UltraRM Reward Model: Designed to distinguish the quality of different answers to the same question. On four public preference test sets, UltraRM significantly outperforms other open-source reward models, achieving SOTA performance.
• UltraCM Critique Model: Used to automatically generate textual feedback. On 9 benchmark datasets, UltraCM outperforms all open-source baselines, with performance approaching ChatGPT.
• UltraLM Language Model: Trained on UltraChat data, possessing rich world knowledge and exceptional instruction understanding and following capabilities. UltraLM-13B-v1.0 topped the Stanford AlpacaEval open-source model leaderboard upon release, while UltraLM-13B-v2.0 (best-of-16 sampling) achieved 92.30% on AlpacaEval, becoming the highest-scoring model under 70B.
Advancing Global Large Model Alignment Research
In the alignment technology pipeline, preference alignment is crucial. Traditional preference alignment data relies on manual annotation, with leading companies collecting hundreds of thousands to millions of preference data points, yet rarely open-sourcing them. Due to the lack of high-quality, publicly available preference datasets, the open-source community remains behind in RLHF research and practice.
ModelBest Ultra alignment has made positive contributions to the global open-source community and received widespread acclaim. Hugging Face's 2023 annual summary highlighted the "ModelBest Ultra Alignment Datasets" from the OpenBMB open-source community as an important milestone in the community's development.
NLP expert and HuggingFace co-founder Thomas Wolf posted a lengthy thread on Twitter, narrating a cross-continental open-source collaboration story involving Mistral + OpenBMB + HuggingFace. The ModelBest Ultra alignment datasets played a key role in the Zephyr model's success story. The H4 team researching model fine-tuning and alignment stated: The new model trained using OpenBMB's two datasets is exceptionally strong, the most powerful model the H4 team has seen in Berkeley and Stanford benchmarks.
This created an open-source legend. As Thomas Wolf said, open access to knowledge, models, research, and datasets worldwide, and people building upon each other's work in AI, collectively creating truly valuable and efficient open models.