Without external training data, how can large models develop reasoning on their own? CPMöbius proposes a Coach–Player collaborative RL framework

2026-05-14

In recent years, large language models have made continuous progress on tasks such as mathematical reasoning, code generation, and complex problem solving. An important reason is that the post-training stage has introduced large amounts of high-quality reasoning data and reinforcement learning signals.

But a problem follows: the stronger the model, the harder it is to obtain the data needed to improve it further.

In domains such as math and code, we can construct reward signals from reference answers, unit tests, or rule-based checks. But as model capability keeps rising, human-curated data that is high-quality, appropriately difficult, and sufficiently broad becomes scarcer, and the cost of construction grows. In other words, current gains in reasoning capability are still largely limited by the data and supervision that humans provide. This raises an important question: without external training data, can a large model continuously improve its own reasoning through autonomous interaction?

CPMöbius, proposed by researchers from the THUNLP lab, a community member, is a Coach–Player collaborative data-free reinforcement learning framework, and the work has been accepted to ICML 2026. In simple terms, a Coach model generates problems based on the Player's current ability, the Player performs reinforcement learning by solving them, and the Coach updates its problem-generation strategy based on whether the Player has actually improved. The paper's abstract notes that CPMöbius does not rely on external training data during the Coach–Player co-evolution stage and surpasses existing unsupervised reinforcement learning methods on multiple mathematical reasoning benchmarks.

The mainstream path to improving large model reasoning usually involves two steps:

SFT, which uses large amounts of high-quality reasoning problems and reference answers to teach the model the problem-solving process.

RL, which further optimizes model outputs through verifiable rewards, preference rewards, or rule-based rewards.

This path is very effective, but it has an obvious bottleneck: it depends heavily on external data.

For mathematical reasoning, high-quality problems and solutions are not unlimited. In more complex reasoning scenarios, manually constructing data is not only costly but also makes it hard to ensure that the difficulty distribution always suits the model's current ability. The paper also notes that the supervision paradigm relying on large-scale human-constructed tasks and labels faces long-term scalability challenges. A new direction is therefore drawing attention: can the model generate its own training tasks and improve its capabilities through autonomous interaction?

This is the problem Data-Free RL aims to solve.

Existing self-play methods reduce the reliance on external data, but they can run into a problem: to "stump" the solver, the problem-setter may generate tasks that are too hard, meaningless, malformed, or unlearnable, leading to unstable training. The paper also notes that adversarial self-play can degenerate into meaningless or unlearnable task generation.

➤ Paper

🔗 https://arxiv.org/pdf/2602.02979

➤ Code

🔗 https://github.com/thunlp/CPMobius

The core idea of CPMöbius: let the Coach teach the Player, not stump it

The core idea of CPMöbius is intuitive: the Coach's goal is not to produce the hardest problems, but the problems that best help the Player improve.

The framework has two collaborating roles:

Coach is the curriculum designer. It generates a batch of mathematical reasoning tasks suited to the Player's current ability.

Player is the reasoning learner. It attempts to solve the tasks generated by the Coach and updates its reasoning ability through reinforcement learning.

The key difference is that the Coach is not rewarded for "difficult problems" but because the Player actually gets stronger after learning from those tasks. The paper designs Coach and Player as two independent but cooperative roles: the Coach generates instructions oriented to the Player's current ability and is rewarded based on changes in the Player's performance; the Player improves its reasoning by solving the tasks the Coach generates. This design turns the "adversarial problem-setting" of traditional self-play into "instructional problem-setting." In other words, CPMöbius is more like a coach training an athlete:

The Coach observes the Player's current level;

the Coach designs training tasks suited to the current stage;

the Player completes the training and improves;

the Coach adjusts the next round of training based on the results.

The paper explains that "Möbius" in CPMöbius refers precisely to this relationship: the Coach and the Player form a continuous loop in which task generation, capability improvement, and feedback updates are continually connected.

CPMöbius turns Coach–Player into a "self-adjusting" learning loop

The CPMöbius training process can be summarized in four steps.

Step 1: the Coach designs training tasks

The Coach first generates a batch of candidate math problems. To avoid problems that are too hard or too easy, CPMöbius introduces difficulty-filtered batching — a difficulty filtering mechanism.

Specifically, the Player first attempts to answer these candidate problems. If a problem is too easy, the Player almost always gets it right; if it is too hard, the Player almost never does. Neither type is suitable for training. The paper uses a lightweight difficulty check to retain problems in an appropriate difficulty range, so that tasks are challenging yet still learnable.

Step 2: the Player solves problems and constructs pseudo-labels

For each task generated by the Coach, the Player samples multiple times to produce several candidate answers. The system then obtains pseudo-labels through majority voting and constructs the reward signal from them.

The advantage is that even without external reference answers, the model can use the consistency across multiple samples to construct approximate verifiable feedback.

Step 3: the Player updates with GRPO

The Player performs reinforcement learning updates with GRPO. GRPO is a critic-free reinforcement learning method suited to estimating relative advantages among multiple candidate outputs. In the paper, the Player updates using tasks generated by the Coach and reward signals formed through majority voting.

Step 4: reward the Coach with the Player's progress

This is the most critical part of CPMöbius.

After updating, the Player is evaluated on a fixed validation environment. The system compares the Player's performance before and after the update. If performance improves, the tasks generated by the Coach were indeed helpful; if not, their teaching value is limited.

The Coach's reward therefore comes from changes in the Player's ability. The paper explicitly states that the Coach receives a scalar environment feedback signal that captures the Player's performance change after the update; the Coach's optimization objective is to generate tasks that bring real learning progress.

This makes CPMöbius more than "a model setting problems for itself"; it forms a more refined learning mechanism:

Whether a task has value is not about how difficult it is, but about whether it makes the learner improve.

Experimental results and ablation analysis: does CPMöbius really bring steady gains?

To validate the effectiveness of CPMöbius, the paper conducts systematic experiments on multiple mathematical reasoning benchmarks, including AMC, AIME 2024, AIME 2025, Minerva, MATH-500, and Olympiad-Bench. These benchmarks cover mathematical reasoning tasks of varying difficulty and problem types. AMC serves as the held-out validation environment during training, measuring changes in the Player's ability before and after each training round, while the other benchmarks are used more to assess whether the model generalizes better. In other words, the experiments care not only about whether the model gets stronger in the training feedback environment, but also about whether that strength transfers to other, more challenging mathematical reasoning tasks.

The paper experiments with four models of differing base capability and training background: Qwen2.5-Math-1.5B, OpenMath-Nemotron-1.5B, OctoThinker-3B-Hybrid-Zero, and Qwen2.5-Math-7B-Instruct.

These models differ considerably in capability: some are basic math models, some have been trained on stronger math data, and some already possess a degree of RL reasoning ability. This setup tests whether CPMöbius works only for a certain class of models or has broader applicability.

The main results show that CPMöbius achieves steady improvements on all four base models. In particular, on the two aggregate metrics overall average and OOD average, CPMöbius generally outperforms the base models and, in multiple settings, exceeds baselines such as R-Zero and RENT, representative unsupervised reasoning training and self-play reasoning training methods. These results indicate that CPMöbius does not merely "game" a fixed validation set but helps the model learn more transferable reasoning ability to some extent. The advantage of CPMöbius is not simply that it "generates more problems," but that it changes the goal of task generation: the Coach aims not to produce harder problems but to produce problems that let the Player improve.

The most direct dynamic signal of Player training

Beyond final results, the dynamics during training also matter. The core mechanism of CPMöbius is that the Coach generates tasks, the Player learns from them, and changes in the Player's ability on the validation environment in turn reward the Coach. So if this loop really works, we should observe one phenomenon: as training progresses, the Player's performance on the AMC 2023 validation environment improves step by step.

The training dynamics curves show that all four base models exhibit upward trends of varying degrees during CPMöbius training. For Qwen2.5-Math-1.5B, AMC 2023 avg@10 accuracy rises steadily from about 0.355 to above 0.39. For Qwen2.5-Math-7B-Instruct, accuracy also rises from about 0.467 to above 0.50. The curves for OpenMath-Nemotron-1.5B and OctoThinker-3B-Hybrid-Zero fluctuate somewhat but still show an overall upward training trend.

This shows that the gains from CPMöbius are not confined to a single evaluation at the end of training but accumulate gradually during training. More importantly, the trend matches the method design: the Coach receives feedback from changes in the Player's performance, and the Player continues learning from the tasks the Coach generates, forming a loop of continuous improvement.

CPMöbius is not simply making the model "more confident"

It is worth noting that improvement on reasoning tasks does not necessarily mean outputs become more consistent. A model may generate the same answer repeatedly, yet that answer is not necessarily correct; conversely, a drop in answer consistency does not necessarily mean reasoning ability has declined — it may mean the model is exploring more different reasoning paths.

The paper therefore further examines Answer Consistency during training. As the figure shows, the answer consistency curves differ across models: for example, Qwen2.5-Math-1.5B gradually declines from about 0.53 to about 0.46; OpenMath-Nemotron-1.5B also trends downward overall; OctoThinker-3B-Hybrid-Zero declines early on and then gradually recovers; and Qwen2.5-Math-7B-Instruct fluctuates without rising monotonically.

This phenomenon is interesting. It shows that the performance gains from CPMöbius do not simply come from "more consistent model outputs" or "a more conservative model." If only answer consistency increased, the model might just output the same kind of answer more often without truly improving its reasoning.

On the contrary, combined with the AMC accuracy curves above, we can see that even when answer consistency declines for some models, AMC validation accuracy still rises. This suggests that CPMöbius is more likely improving the quality of the model's reasoning rather than simply increasing answer repetition.

In other words, what CPMöbius brings is not simple confidence calibration, but real capability gains for the Player from more valuable training tasks through Coach–Player interaction.

Which components really matter?

CPMöbius raises three core questions: 1. Does the Coach need to learn dynamically? 2. Does the Coach need basic problem-generation ability? 3. Do generated tasks need difficulty control?

1. The Coach must update dynamically

If Coach Update is removed, the Coach degenerates into a relatively static task generator. It cannot adjust tasks according to changes in the Player's ability, making a genuine curriculum learning process hard to form.

Ablation results show that on Qwen2.5-Math-1.5B, full CPMöbius reaches an average of 28.8 and an OOD average of 26.8; removing Coach Update drops the average to 25.3 and the OOD average to 23.1. This shows that the Coach's value is not just "being able to generate problems" but being able to continuously adjust its problem-generation strategy based on the Player's learning results. That is, the core of CPMöbius is not static synthetic data but dynamic teacher-student interaction.

2. Coach Warm-up helps the system start more stably

The paper also analyzes the role of Coach Warm-up. Intuitively, if the Coach cannot generate high-quality math problems at all in the beginning, subsequent training can easily fall into a loop of low-quality tasks.

For example, the Coach may generate malformed problems, problems without a clear answer, or tasks with little relevance to improving mathematical reasoning. In that case, even with reinforcement learning, the Player struggles to obtain effective training signals. Ablation results show that after removing Coach Warm-up, the average on Qwen2.5-Math-1.5B drops to 23.7 and the OOD average to 21.2, a clear decline compared with full CPMöbius. This shows that although CPMöbius emphasizes data-free co-evolution, giving the Coach basic task-generation ability before entering co-evolution still matters. In other words, the Coach does not need external data for continuous training, but it needs at least a good enough starting point to enter an effective self-improvement loop.

3. Instruction Filter keeps tasks in the "learnable zone"

The third key component is the Instruction Filter, which can also be understood as a task-difficulty filtering mechanism.

In CPMöbius, not all tasks generated by the Coach are used directly to train the Player. The system judges task difficulty from the Player's initial problem-solving performance. If a task is too easy and the Player almost always gets it right, it offers limited training value; if a task is too hard and the Player can almost never solve it, it cannot provide an effective learning signal either.

Truly valuable tasks tend to lie in the middle: challenging but still learnable. Ablation results show that removing the Instruction Filter drops the average on Qwen2.5-Math-1.5B to 24.9 and the OOD average to 22.5. This shows that difficulty control is especially important for data-free training, because without human data curation, the quality and difficulty of tasks the model generates itself can be highly unstable. Without filtering, training data would be mixed with large numbers of overly easy, overly hard, or invalid samples, ultimately hurting the Player's learning efficiency.

Conclusion

We have grown used to letting models generate their own data and take it for granted that as long as there are enough problems and they are hard enough, the student model will get stronger. CPMöbius's answer is: not necessarily. Hard problems do not automatically equal effective training, and generating more problems does not automatically equal better reasoning. What really matters is not whether the Coach can "stump" the Player, but whether the problems it produces land exactly where the Player can currently learn; whether the training signals it provides actually lead to capability gains; and whether it can keep adjusting the next round of curriculum based on the Player's progress.

And this may be where CPMöbius is most valuable: it does not reduce data-free reasoning training to "letting the model make its own problems," but asks further why some tasks help a model improve while others are merely harder. It reminds us that future self-improvement of large models may not be about "generating more data" but about designing a better learning relationship: making the Coach not just stronger but better at teaching, and the Player not just solve more problems but truly learn to reason.