Tsinghua team revisits OPD: data is stuffed, the algorithm starves

2026-10-12

This work is a collaboration led by Tsinghua University with researchers from the University of Chinese Academy of Sciences, Northeastern University, the University of Illinois Urbana-Champaign, Johns Hopkins University and other institutions.

What happens if you cut the training set for On-Policy Distillation (OPD) from 17,000 problems down to one?

Intuition says the model will master that single problem quickly and the curve should flatten within a few dozen steps. That is not what happens: it keeps climbing for hundreds of steps without stopping. The average score on three math benchmarks starts at 59.1 and reaches 68.5 at 300 steps, while the control trained on all 17,000 problems sits at 69.8 at the same point — a single data point has already captured most of the full-data gain.

This is the second post in the Rethinking On-Policy Distillation series. The first asked why OPD works and concluded that teacher and student reasoning patterns need to match, and that a higher score does not mean there is something new to teach. This one pushes the question further: given only a single sample, why can OPD still improve steadily over hundreds of steps and approach the effect of the full dataset?

The Tsinghua team's answer is blunt: data is stuffed, the algorithm starves.

🎯 Core finding: the rollout of one problem already touches 71.5% of the full state space; 16 semantically different problems reach 98.9%

🤔 Surprise: the absorption-rate decay curve is almost identical whether you feed 1 problem or 17,000

💡 Key takeaway: what holds OPD back is not a lack of data but how slowly the algorithm digests it

➤ Paper

🔗 https://huggingface.co/papers/2609.04172

➤ Code repository

🔗 https://github.com/Thinking-Space/One-Shot-OPD

Background: OPD's algorithm has been studied repeatedly, the data side rarely measured

Over the past year, OPD has become a standard component of post-training for frontier models: the student samples on its own, the teacher provides a full target distribution at every token position the student visits, combining on-policy state visitation with dense token-level supervision.

There has been a good deal of mechanistic work on it, but almost all of it stands on the algorithm side — the training set is treated as a given. On the data side, the usual move is to add: collect more, clean more, synthesize more.

The Tsinghua team asks the other half of the question: if the training set shrinks to a single problem, what can the model still learn?

Approach: cutting the training set from 17,000 problems to one

On RLVR, one-shot RLVR has already run controlled experiments with "keep only one problem in the training set", and that work directly inspired this one. It is worth testing again on a different object because the two methods place their signal differently: RLVR's signal lands at the end of the trajectory, while OPD's is spread across every visited position.

The experiment keeps every other setting unchanged and shrinks only the math training set from DAPO-Math-17K to a single problem, then tracks both the training curve and internal metrics, extending the conclusion to other task domains, other teacher-student families and multi-teacher settings.

Main result: one problem, hundreds of steps of steady gains

On math, the student's initial average score is 59.1. By 300 steps the single-problem setting reaches 68.5 and the 17,000-problem control reaches 69.8 — the single problem captures 87% of the full OPD gain and closes 69% of the teacher-student gap. Running on to 1,000 steps, the single-problem setting is at 68.4 and the full control at 72.1, with the recovery ratio falling to 72%. After 300 steps, though, the two curves hold a gap of roughly 3 points. In other words, one problem captures most of the full OPD gain, but training longer does not close the gap to the full dataset.

The phenomenon holds in other task domains and teacher-student families too: moving to coding, instruction following and agentic tool use, single-problem training closes 73%, 66% and 64% of the initial teacher-student gap respectively; across the Qwen, Llama and OLMo teacher-student pairs, One-Shot OPD also delivers substantial gains.

It is also almost insensitive to the nature of the training problem itself. Three math problems of different difficulty give similar training gains, including the hardest one, which the student never solves even once in 300 steps while its validation performance still improves; changing the response length cap and sampling temperature does not change the conclusion either.

Two questions remain: with only one training input, why does the model still gain so much? And why do those gains take hundreds of steps to form?

Data side: one problem's rollout touches 70% of the state space

To answer the first question, look at OPD's unit of learning. It is not the problem but the state: a query plus the half-written answer the student has produced so far, forming the situation it faces at the next step, each with a target distribution supplied by the teacher.

A problem therefore supplies more than one piece of supervision. Under default settings each update contains 64 rollouts, so even with a single problem, one update produces tens of thousands of supervised token positions. The number of problems records how much input there is, but says nothing about where those inputs take the model.

To quantify how broadly states are covered, the researchers introduce a metric called "state coverage": the states visited by full-data OPD are pooled into a reference set and clustered into 200 clusters, and a setting's coverage is the share of those 200 reference clusters it reaches.

It turns out one problem already touches most of the full state space, and most of that coverage happens within the first few dozen steps. Coverage is 65.9% at 100 steps and 71.5% at 300, with only 5.6 more points over the next 200 steps. New states are still being produced — resampling the same problem reaches a wider state space — but the rate of new ones slows markedly.

Coverage and validation performance rising together only shows correlation. The researchers ran control experiments on both the rollout-trajectory side and the problem-query side.

On the trajectory side, the initial student generates 64 fixed trajectories for the same problem, and 1, 4, 16 or all 64 are kept and used to repeatedly fill a training batch of the same size: the more kept, the higher the validation accuracy, rising from 0.606 to 0.666.

On the problem side, taking one representative problem from each of several semantic clusters gives training sets of 1, 4 and 16 problems: state coverage comes out at 71.5%, 79.8% and 98.9% respectively, and under the same training budget the 16-problem result is essentially on par with the full dataset.

Equal numbers of problems can still differ enormously in diversity. The researchers compared two 16-problem training sets: one drawn from 16 semantic clusters, the other entirely from a single cluster. The former is higher in both coverage and validation performance; the latter reaches about 76.8% coverage in the first 100 steps or so and then barely grows.

These results point to one explanation: in OPD, the value of a problem depends largely on whether it can carry the student into states it has not visited before. A single input is small, but repeated rollouts produce many states; add problems that are semantically spread out and coverage can expand further. In this paper's math experiments, throwing the entire DAPO-Math-17K into training brought no more visible benefit than these 16 semantically diverse problems. Relative to the amount of supervision current training can use, the data is already stuffed.

Algorithm side: how many problems you feed barely affects the absorption rate

Coverage answers "how much supervision is available", not "how fast the student uses it". The paper therefore defines two training-dynamics metrics, one describing how far it has eaten and one how fast it eats: distance d is the mean absolute difference between the teacher's and student's log probabilities for the actually generated token at the student's current visited position; absorption rate v is the fraction of the remaining distance removed by one update.

First observation: the student never stops; each step just makes less progress. Distance falls throughout the run. What actually declines is the absorption rate — which is why a run needs hundreds of steps rather than dozens.

The second observation is the key one: this declining curve barely changes with training set size. Across the 1-, 4-, 16- and 17,000-problem runs, by 300 steps each has removed 78% to 84% of its own distance. In other words, adding problems does not make alignment faster, and removing them does not make it slower: the pace of alignment is set more by the OPD algorithm itself than by the number of problems.

Another explanation would be that on-policy training keeps producing new states, the model has to keep adapting to changing data, and that is what stretches the run. The researchers added a fixed-trajectory control: always reuse the 64 rollouts generated by the initial student, never sampling new states from the current model. Even with training states completely fixed, validation performance still improved steadily for about 200 steps. A continuous supply of new states is not the only reason for the long training curve; absorbing the supervision on existing states itself takes many updates.

Could the learning rate be the cause? The researchers tried 5×10⁻⁷, 1×10⁻⁶ and 2×10⁻⁶, a fourfold range: raising it lets the model complete a similar alignment process in fewer steps, but rescaling the horizontal axis by learning rate makes the three distance curves essentially coincide. Within the tested range, the learning rate changes the timescale, not the overall shape of the declining absorption speed.

Put the two sides together: the data side explains why a single problem captures a large gain — repeated rollouts over one problem already cover a wide enough set of states. The algorithm side explains why training still takes hundreds of steps — as alignment proceeds, each update can absorb less and less of the remaining supervision, and adding more problems does not visibly change that. Together they make up what the paper calls "data is stuffed, the algorithm starves."

Extending to multi-teacher OPD: a dozen problems per domain match the full set

In practice, Multi-teacher OPD (MOPD) is more common: several domains go into one training run, each supervised by its own teacher, and each problem routes to the teacher of its domain. The researchers ran the same ladder, and the conclusion holds — in this three-domain, one-teacher-per-domain setting, a dozen or so semantically distinct problems per domain match full MOPD.

The student's initial average validation score is 43.5. At 300 steps, 1 problem per domain gives 50.1, 16 semantically different problems per domain gives 52.9, and full MOPD gives 52.8, with the average across the three teachers at 55.4. How far the conclusion extends as the number of teachers grows further is not tested in this paper.

Extreme experiment: even "non-problems" can train the model

If the explanation "states are the unit of learning" holds, it yields a somewhat radical prediction: the task content in the input may not be essential. The researchers designed three stress tests: an empty template (empty user turn, ending with only a

), System Template (a single line of domain instruction added), and WildChat prompts (192,824 real user chats, of which only 0.17% are math-related under a conservative judgment).

All three conditions stay close to the real-data baseline while consuming only one-third to one-half of its rollout tokens. In the code domain, over 500 steps real data consumes 277M rollout tokens while the empty template consumes only 18M.

But not just any text works as input. If the template closes the

block right at the start, the student produces overly short meta-level replies and no usable training trajectory forms.

So the more accurate statement is: task content is not the only source of valid supervision in OPD input; the input also determines where the student starts writing and whether the rollout can reach a state where the teacher has something to teach. When evaluating an input, look not only at what task it poses but at what states it can elicit.

Finally, the researchers ran a controlled comparison of OPD and RLVR on the same problem: over 1,000 steps, OPD's validation gain is more than twice RLVR's, while closing 72% of its own gap to the teacher. But the two ceilings point in different directions — RLVR is not constrained by any teacher distribution, while OPD aims to match one. This is not "OPD is better than RLVR"; each is held back by something different: one by how much outcome variation the data can supply, the other by how fast it absorbs dense supervision.

Closing: how much you feed vs. how much gets digested

None of this says training data does not matter. It says that in OPD, the number of problems may not be a suitable measure of how much valid supervision there is: a single problem can produce a large number of different states through continued rollout, so with very few input problems the student still gets a rich enough training signal. What matters more than the problem count is which states these problems ultimately lead the model into, and how much of the effective space those states cover.

This also offers a different angle on data selection: to evaluate a training problem in OPD, the question may not be how hard or how clean it is, but whether it can take the student somewhere it has not been, whether the teacher still has something to teach there, and whether, at the student's current absorption rate, that supervision can be digested within a few hundred steps.

Taken together, the two posts in the Rethinking OPD series point to the same thing: dense rewards in OPD are not a free lunch, and what is genuinely scarce is often not the supply. The first post showed that a teacher's higher score does not mean it teaches better; this one shows that feeding more data does not mean the student can digest it right now.

The boundaries deserve stating clearly. State coverage is a semantic-level proxy: the reference space is built from one full-data run, so it measures what fraction of the reference space a setting reaches, not that setting's own coverage; it weights all clusters equally, distinguishing neither visit frequency nor how much teacher-student difference remains in each cluster. Why the absorption rate declines with training is also not settled. Both points are noted in the paper's Limitations.

For anyone working on OPD, this conclusion has a cheap use: swap the problem set for a dozen semantically dissimilar problems and run it alongside your full-data run for a few hundred steps. If it still matches after the cut, the bottleneck really is not the number of problems; if it drops by a chunk, this set has not covered enough states. Either result is a net gain.

The old saying about biting off more than you can chew applies to models as well.

See the original paper and open-source repository for more details.