ICLR 2026 | How far can unsupervised RL for large models go? A Tsinghua team gives a systematic answer

2026-05-23

The next stop for RL: from supervised to unsupervised

Reinforcement learning is reshaping the capability frontier of large models. Leading models such as OpenAI o3, DeepSeek-R1, and Gemini 3 are all using large-scale RLVR (reinforcement learning with verifiable rewards) to push the ceiling of reasoning tasks. But everyone knows that purely supervised training is not sustainable. The cost of human annotation is growing exponentially, and reliable labels are increasingly hard to obtain in specialized domains. As model capability approaches or even surpasses human experts, who will grade the model?

Starting with TTRL, unsupervised RLVR entered researchers' field of view, with the goal of letting models keep evolving without human annotation. This is not only a matter of cutting costs and improving efficiency, but a necessary path toward superintelligence. Just as pre-training trained GPT on unlabeled data, does unsupervised RLVR hold similar potential? This paper attempts to offer a preliminary answer.

➤ Paper

🔗 https://arxiv.org/abs/2603.08660

➤ GitHub

🔗 https://github.com/PRIME-RL/TTRL/tree/urlvr-dev

➤ X Thread

🔗 https://x.com/HBX_hbx/status/2031406636930338828

In a recent study, researchers from the THUNLP lab, a community member, drew the first boundary on this seemingly promising picture. They systematically dissected the inner workings of unsupervised RLVR and found that all intrinsic-reward methods based on the model's own signals — whether majority voting, entropy rewards, or other variants — follow a similar trajectory: performance climbs rapidly early in training, but after reaching a critical point, it begins an irreversible slide. This is not a flaw of any particular method but the fate of the mechanism: they are all essentially sharpening the model's existing preferences, like an echo chamber that makes the model repeat what it initially believed. If the initial confidence happens to be correct, the results are striking; if it is mismatched, collapse is only a matter of time.

But that does not mean intrinsic rewards are worthless. In small-scale test-time training, they can still reliably improve performance; even if the model is wrong from the start, it can evolve through self-correction. More importantly, the researchers found a "predictive indicator" that can forecast a model's trainability before large-scale training, without running the entire curve.

When intrinsic rewards are limited by the model's own echo, extrinsic-reward methods begin to paint a different picture — for example, letting the model exploit the asymmetry between generation and verification to anchor its reward. These methods are breaking through the ceiling of intrinsic rewards and moving unsupervised RL toward true scalability.

The core judgment this paper conveys is: on the road to superintelligence, the key is not to blindly believe that models can self-evolve, but to know when to let them listen to their own echo and when to push them toward real-world verification.

A taxonomy of unsupervised RLVR methods and an overview of the paper's key findings

Intrinsic reward methods: deep problems beneath a flourishing surface

Over the past year, various "intrinsic reward" methods have emerged in quick succession. From majority voting to variants based on the model's confidence or entropy, they use the model's internal signals to construct a proxy reward, and performance soars early in training, sometimes even surpassing supervised methods.

The researchers group these methods into two categories by reward source: one based on certainty, which directly takes a confidence metric over the model's reasoning traces as the reward; the other based on ensembles, which anchors correctness using the aggregated result of multiple rollouts (such as majority voting).

The reward is free, but the price is steep. After the early surge in performance, continued training triggers classic reward hacking:

the proxy reward keeps rising while real performance collapses

the model grows more confident while its answers grow more absurd

different intrinsic reward methods behave wildly differently across models

More critically, no one can clearly explain why it works or why it fails.

What this study did: opening the black box and drawing the boundary

The authors say they did not want to simply "propose a new method and chase a higher number," but to answer the question no one had clearly settled:

Where is the scaling ceiling of unsupervised RLVR? And if there is a ceiling, where is the boundary?

To that end, the researchers did five things:

Unified theoretical framework: They subsumed seemingly diverse intrinsic reward methods under a single mechanism, revealing their common essence — sharpening the model's initial distribution — and derived a theoretical convergence boundary.

Large-scale empirical study: 11 models × 5 intrinsic reward methods × hyperparameter sweeps. The data shows that "rise then fall" is not a coincidence but a general law.

Mapping the safe zone: Not every setting collapses. The researchers found that in small-scale test-time training, intrinsic rewards can be used safely, and the model can evolve reliably even when it starts out entirely wrong.

Turning the trap into a signpost: Rise and fall is not only a risk; it is itself information. The paper uses this phenomenon to derive a model-prior indicator that predicts whether a base model is suited to reinforcement learning without running the full RL curve.

Exploring alternatives: Since intrinsic rewards have a ceiling, the researchers turned their attention outward. They made a preliminary exploration of extrinsic reward methods based on generation-verification asymmetry, to see whether these can truly break the scaling limit of intrinsic rewards.

Four key findings: 🔍 Finding 1: success hinges on "confidence-correctness" alignment

The researchers built a unified theory of intrinsic reward methods, revealing their shared essence: sharpening the distribution — amplifying the model's existing preferences rather than creating new knowledge. This mechanism has a property:

If the model's initial inclination is correct → sharpening works, and performance improves

If the model's initial inclination is wrong → sharpening is harmful and accelerates collapse

The paper defines the model's initial inclination (or model prior) as the degree of confidence-correctness alignment: when only the model's self-consistency is improved, how likely is it to directly get more problems right. In other words, a model with a strong prior already possesses most of the knowledge needed to solve the problem, but lacks the confidence to state the correct answer.

The researchers tested 11 models, 5 methods, and 4 commonly used hyperparameters, and the conclusion seems harsh: collapse is inevitable, only a matter of time. Even the most stable configuration cannot last more than a few epochs. This suggests it may not be an engineering problem but a mathematical necessity.

Left: success depends on the degree of confidence-correctness alignment; right: the evolution of confidence and correctness on a single data point during training

✅ Finding 2: small settings are actually safe

Rise and fall is fate, but fate has its scope.

When training data is scarce enough — for example, in domain-specific scenarios like test-time training — intrinsic reward methods show rare stability. The reason is simple: when confidence is optimized on only a handful of samples, the model runs out of room quickly. Even if it becomes "super confident" on those samples, it is hard for that to trigger a global policy shift, and accuracy on OOD tasks holds steady.

More interesting is an extreme experiment: the researchers deliberately chose 32 samples that the model got entirely wrong as the training set. In other words, the proxy reward from intrinsic rewards was wrong from the very beginning. The result? Performance on the OOD test set still improved steadily.

The researchers therefore point out that intrinsic rewards are not teaching the model "what is correct" but teaching it to "believe in itself more." Even when that belief is misplaced, the self-reinforcement stays locked in a local scope and cannot stir up a large wave.

Left: small-scale TTT improves steadily without collapse; right: policy KL divergence under different training-set sizes

🎯 Finding 3: judging whether a model is suited to RL

Rise and fall is not just a risk; it is information in itself.

Since the success of intrinsic rewards depends on the model's initial "confidence-correctness" alignment, can this alignment be used to judge in advance whether a base model is suited to RL? After all, running large-scale RL is expensive, and the field has long lacked a lightweight predictive indicator.

The researchers found a ruler: Model Collapse Step, which measures how many steps a model can last under intrinsic-reward training before it fully collapses. The logic is simple: the later the collapse, the better the model's initial prior — it already holds more correct knowledge but is not confident enough; and this prior is exactly what standard supervised RL can amplify. In other words, the collapse point of intrinsic rewards is a natural indicator of a model's "RL trainability."

The results confirm this. The Qwen family, widely regarded as "well suited to RL," lasts longer under intrinsic rewards. More interestingly, the indicator requires no ground-truth labels, and its prediction accuracy exceeds the traditional pass@k.

The paper thus conveys a methodological insight: turn failure into a signpost, and turn expensive trial and error into a lightweight prediction.

Left: model collapse steps of different base models under unsupervised intrinsic-reward training; middle: the performance gain of the corresponding base models under supervised RLVR. The later the collapse under unsupervised intrinsic rewards, the better the result after supervised RLVR — with prediction accuracy exceeding the traditional pass@k.

🚀 Finding 4: extrinsic rewards are the scalable direction

If intrinsic rewards are destined to have a ceiling, where is the way forward?

The root of the problem lies in where the reward comes from. Intrinsic reward methods use the model's own confidence to train the model itself — a closed-loop echo chamber where the reward signal is forever limited to what the model already knows. You cannot use it to teach the model knowledge it truly does not have.

But unsupervised RLVR goes beyond this. The paper groups extrinsic reward methods into two categories:

Exploiting unlabeled data: mining reward signals from massive corpora. The more data, the richer the reward signal, and it does not run dry as the model gets stronger.

Exploiting generation-verification asymmetry: let the model generate answers, then use external tools (compilers, proof assistants, simulators) to verify them and provide environment feedback. These verifiers do not stop working as the model gets stronger; their judgments remain objective.

The researchers ran a preliminary test of self-verification methods, and the result was a completely different curve: continuous improvement with no collapse. The reason is simple — the reward does not come from "how confident the model is" but from "whether the answer passes objective verification." Coming up with a solution may be hard, but checking whether it is right is often easy; this asymmetry anchors the model's evolution to the iron laws of the real world rather than its own echo.

Intrinsic rewards ask, "Do you believe in yourself?" Extrinsic rewards ask, "Is it true?" The answer to scalable unsupervised reinforcement learning may lie in the latter.

Some tasks exhibit generation-verification asymmetry

Closing: beyond the boundary

This paper spends many pages mapping the boundary of unsupervised reinforcement learning. But the value of such a map has never been in telling you "this road is closed," but in answering: under what conditions, which road is open.

Whether a system can become better by examining itself depends on how accurate its initial judgments are. The reason intrinsic reward methods fail is exactly the reason they succeed — the same mechanism: self-reinforcement. The only difference is whether what gets reinforced is truth or bias.

In the authors' view, only by recognizing the fate of intrinsic rewards can we truly see the vast sea of stars that is extrinsic reward. The path to scalable unsupervised reinforcement learning requires not a blind belief that models can self-evolve, but the knowledge of when to let them listen to their own echo and when to push them toward real-world verification.

What the paper ultimately wants to convey is that intrinsic and extrinsic are not opposites but different tools in the same toolbox. Recognizing the boundary is not about stopping, but about creating freely within it and seeking new possibilities beyond it.