The same textbooks yield only a spoonful: Tsinghua’s StudyBench measures self-evolution efficiency
While the whole industry bets on self-evolution and writes RSI (recursive self-improvement) into its roadmaps, one plain question is rarely quantified: given the same raw material, how much of that knowledge can a model actually read into capability?
The Tsinghua team's answer, via StudyBench, is blunt: the knowledge is in the book; what's missing is a method to squeeze it out.
🎯 Key finding: textbooks in context score full marks; self-training methods extract only single digits
🤔 Surprise: even at 600+ GPU-hours, performance plateaus early
💡 Takeaway: what's missing is neither knowledge nor compute, but the method that turns knowledge into capability
The StudyBench pipeline: from textbook sources through extraction and filtering to the two test surfaces, absorption and transfer
➤ Paper
🔗 https://arxiv.org/abs/2609.00787
➤ Code
🔗 https://github.com/thunlp/StudyBench
Background: everyone is betting on self-evolution, but no one has measured how much capability it produces
Lately, Anthropic has talked up the idea that 80% of code can be handed to models that improve themselves; OpenAI has sketched self-evolution for the next GPT; Zhipu has said the next GLM will take the same path. RSI (recursive self-improvement) has gone from science fiction to a goal written seriously into roadmaps.
The logic is simple and the stakes are large: if a model can turn raw material into capability and keep improving without more human labeling, the slope of that curve decides almost everything. Methods arrive one after another, and the scores in the reports keep climbing.
In that optimism, one question hides behind the phrase "score gains": doing the exercises in the book and doing harder problems beyond it are not the same measurement. Is the gain real understanding of the book, or just practice on the problems at hand?
So StudyBench returns to a more fundamental question: what capability does self-evolution actually require? Ultimately, it is absorbing knowledge and then transferring it to harder problems beyond the book. We split the single aggregate score back into absorption and transfer, while controlling for the model's existing ability and differences in training data. We chose physics, a domain that is both verifiable and has a clear difficulty gradient, to measure the gap.
How it's built: 11 books, six competitions, one reachability guarantee
The training set is 11 classic physics textbooks, about 6.0M tokens, from which the team extracted 1,420 problems with answers and 646 without. Physics was chosen because answers are verifiable, difficulty gradients are clear, and the problems test both reasoning and memory.
The Application Set (88 parent problems · 109 subproblems) keeps only in-book hard problems that Qwen3-8B cannot solve reliably. The Transfer Set (90 parent problems · 280 subproblems) comes from six international competitions: IPhO, APhO, IOAA, NBPhO, EuPhO and OPhO.
Crucially, every transfer problem passes a reachability check. A five-step teacher pipeline (decompose, retrieve, guide, verify, retry) attaches solution guidance drawn from the textbook text; a problem enters the Transfer Set only if Qwen3-8B fails it alone but solves it completely at least once in 8 attempts with the guidance. This removes the excuse that the problems are simply too hard for the training set.
The two filters put the benchmark in an honest position: the absorption set excludes freebies the model already solves, and the transfer set excludes problems that cannot be derived from the book. Only with clean test boundaries does the measured gap mean anything. The team also masked parts of the training material so that rote memorization cannot supply answers: a method that memorizes every page still cannot produce any of the answers it will be tested on.
Main result: on Qwen3-8B, absorption rises sharply while transfer barely moves
The best method, GEPA, lifts Qwen3-8B's Par@8 on in-book hard problems (pass@8 on parent problems) from 17.05 to 34.85, nearly double; on competition problems, Transfer Par@8 is only 7.04, still in single digits.
Same method, same training set: absorption improves clearly, transfer stays put. Swap in Llama-3.2-3B-Instruct as the base and it cannot even absorb. The strongest base does not change the story. Opus 4.7 starts at 40.00 on Transfer Par@1, yet most methods finish lower than they started — GEPA reaches 41.11 and EvoSkill 43.33, barely level, while ACE drops to 33.33. Even the strongest base cannot push these competition problems up through self-evolution.
Guidance Gap: with textbook guidance, scores return to near perfect
The Transfer Set consists of problems Qwen3-8B can solve with textbook guidance. That guidance was assembled by DeepSeek-V4-Pro from the textbook text and checked to ensure DeepSeek-V4-Pro does not leak the answer or intermediate results absent from the book.
The researchers ran a direct control: give the same guidance back to models evolved by each method, then rerun the same transfer set. Apart from Bonito, every method scores near perfect. Bonito's regression comes from a training method that damages Qwen3-8B's reasoning ability; even so, it scores high with the guidance.
The team calls this gap the Guidance Gap: the score obtained when the book is handed to the model minus the score a method trains out on its own. The wider the gap, the more the method failed to convert knowledge into capability. However you read the remaining nine-tenths-plus of the gap, one conclusion holds: the problem is neither the training set nor the model's innate ability, but the method meant to turn the training set into capability — it failed to internalize it.
Compute Plateau: more compute brings no further gain
Could it be insufficient compute? We tracked evaluation results across training.
The answer: almost every method plateaus at the first checkpoint; after that the curve flattens or falls.
Swipe left for more
The five curves take different paths but converge on one statement: the training set is not too small and the compute is not too little — the method is wrong. What needs replacing is the self-evolution loop itself, not running the same loop longer.
What this work is saying
This is not a verdict against self-evolution; we still believe it is worth pursuing.
❌ We are not saying current self-evolution methods are useless or that training has no effect. ✅ We are saying their gains mostly stop at the "homework" level and barely transfer to harder problems.
❌ We are not saying the model cannot learn this physics. ✅ We are saying it gets it right (100.00) when the book is in context; the "evolve it yourself" step is what fails.
❌ We are not saying more compute would close the gap. ✅ We are saying adding compute still plateaus early — change the loop, not the training time.
Final thoughts: on squeezing
Perhaps StudyBench's greatest value is turning an open-ended line of research, long judged by "how many points did it gain," into something that can be measured precisely: with the same textbooks, one side is the Olympiad-level capability the raw text already contains, the other is the spoonful a method squeezes out. The gap between them is exposed for the first time. Only what can be measured can be improved.
For those working on self-evolution, the benchmark has two uses. If your method does well on absorption but cannot move transfer, you have located its ceiling. If it turns out to be one of the few that lifts transfer, that is worth writing up. Either result is a net gain.
See the paper and the open-source repository for more details.