Tsinghua Team Reveals in New Paper: Why Does LLM On-Policy Distillation Keep Failing? From Symptoms to Mechanisms to a Rescue Guide

2026-05-02

Is the free lunch of distillation really that good?

In today's post-training pipelines for large models, On-Policy Distillation (OPD) has become a star technique. From Qwen3 and MiMo to GLM-5, the industry has adopted OPD and reported huge performance gains. Compared with the sparse outcome rewards of reinforcement learning (RL), OPD provides dense token-level supervision — it looks like a free lunch.

But if you have run OPD yourself, you may have hit a counterintuitive phenomenon: why does swapping in a stronger teacher leave the student's performance flat, or even worse?

Distillation in the large-model era is no longer a simple matter of brute force producing miracles.

In a new paper, a Tsinghua University team systematically dissects the black box of on-policy distillation. The work reveals two preconditions that determine success, digs into token-level alignment mechanisms, and offers practical recipes for rescuing failed distillation.

📄 Paper

🔗 https://arxiv.org/abs/2604.13016

💻 Code repository

🔗 https://github.com/thunlp/OPD

𝕏 Thread

🔗 https://x.com/HBX_hbx/status/2044464414829777354

Symptoms: Why don't great teachers produce great students?

Conventional wisdom says the higher a teacher model scores, the better the distillation should be. But through rigorous controlled experiments, the Tsinghua team found two core laws that govern OPD's fate:

Law 1: Thinking-pattern consistency

The authors had a weak base model, Qwen3-1.7B-Base, learn from two teachers of similar capability: Qwen3-4B (Non-thinking) and Qwen3-4B-Base-GRPO, which was trained only with GRPO. Because the student is also a base model, its thinking pattern was closer to the GRPO-reinforced base teacher (a higher initial Overlap Ratio), and the final distillation result improved significantly. If thinking patterns mismatch early on, it is very hard to make up for it later.

Law 2: Higher scores ≠ new knowledge

If teacher and student share a thinking pattern and the teacher scores higher, is distillation guaranteed to work?

The authors saw the same phenomenon in both the DeepSeek and Qwen families: a teacher that is merely larger, from the same pipeline and recipe, yields very limited gains; a teacher with additional RL post-training recovers much more of the teacher-student gap. In the DeepSeek family, for example, the RL-trained Skywork-OR1-Math-7B achieves 16.9% gap recovery, while DeepSeek-R1-Distill-7B from the same pipeline reaches only 5.3%; in the Qwen family the gap is as large as 58.6% versus 15.6%.

This suggests that if a teacher is merely a larger version built on the same pipeline, data, and recipe, it may look to the student like "a different-scale version of the same distribution" and provide little new transferable signal.

The most striking experiment: distilling the student back

The authors ran an extreme "reverse distillation" experiment: they took the post-RL JustRL-1.5B as the student and had it learn from its own pre-RL checkpoint, R1-Distill-1.5B, with a larger, higher-scoring R1-Distill-7B from the same family as a control. The result was surprising: learning from the 7B and learning from the 1.5B produced almost the same effect — both rolled the student back to its pre-RL level, with very similar decline curves! So although the 7B scores higher, its advantage over the 1.5B is merely the dividend of parameter scale; it offers the student no more learnable information. OPD is not simply "learning high scores"; it is actively extracting and replicating the teacher's thinking pattern.

Mechanisms: What the token-level microscope reveals

When OPD succeeds or fails, what actually happens at the token level?

The authors monitored dynamic metrics throughout training and found a very clear pattern: successful distillation is a mutual convergence of high-probability tokens.

In successful OPD, the overlap ratio between the student's and teacher's top-k predicted tokens climbs steadily from 72% to above 91%, while the entropy gap between them shrinks rapidly. In failed OPD, these metrics barely change from start to finish.

More importantly: the overlap region is everything.

The authors decomposed the distillation objective and ran a set of ablation experiments. They found that the high-probability tokens favored by both teacher and student are the core engine of optimization, contributing most of the gradient and advantage. Computing the loss only on these overlap tokens barely dents distillation performance! The non-overlapping tokens contribute almost nothing to optimization.

Prescriptions: Two remedies for "hopeless" distillation

If all you have is a teacher whose thinking pattern does not fit, are you out of options? Based on these findings, the team offers two targeted remedies:

1. Off-policy cold start

Since starting directly with on-policy distillation tends to cause thinking-pattern mismatch, force alignment with off-policy first. Before OPD begins, run a lightweight round of SFT for the student on rollouts generated by the teacher. This directly raises the initial overlap ratio, lets the subsequent OPD training start smoothly, and ends at a higher performance ceiling than a pure OPD baseline.

2. Teacher-aligned prompts

Since the teacher's policy was shaped on a certain kind of post-training prompt, expose OPD to prompts closer to the teacher's training distribution, aligning both at the template level and the content level. The paper finds this indeed further improves accuracy and overlap growth, but at the cost of faster student entropy decline, so it is best mixed with some OOD prompts to avoid premature entropy collapse.

Discussion and Reflection: Can OPD scale indefinitely?

A free dense reward signal is tempting, but the authors find that reward signal quality degrades sharply with trajectory depth.

In responses as long as 15K tokens, the team observed a clear "backward entropy collapse": as generation goes deeper, the student's prefix drifts further from the distribution the teacher knows, so the teacher's reward in the latter half becomes pure noise and triggers a collapse of the whole training run. This means OPD is currently hard to extend directly to long chain-of-thought or agentic multi-turn scenarios. There is a fundamental tension between dense supervision and supervision reliability.

Moreover, a globally useful reward does not mean it can be optimized effectively locally. A failing teacher's global reward is actually not weak: its AUROC for distinguishing correct from incorrect rollouts is even close to that of a successful teacher. So the failure is not because the reward signal carries no information, but because the local optimization geometry of the reward is broken — informative globally, yet flat locally.

Final Thoughts: On "Stronger" and "More Learnable"

We are used to finding a larger model to extract knowledge from, assuming that a stronger teacher teaches better. This paper's answer is: not necessarily. In OPD, stronger does not automatically mean better at teaching. A higher score does not automatically mean new knowledge. And a large model does not simply "pour" capability into a small one; it is more like passing on a reasoning path, a way of organizing local preferences.

So the real question is not "how strong is the teacher," but:

Does it speak the same language as the student?

Is what it brings something the student has not truly learned yet?

Can the supervision it provides become useful gradient at the student's current position?

And that may be the paper's most valuable contribution: instead of adding another trick to OPD, it systematically reveals, for the first time, a simple fact — why some teachers can teach students, and why others are merely "stronger in appearance."