1. Introduction

To train a catgirl with reasoning ability, do you absolutely need a stronger, higher-quality catgirl teacher? But where would you even find such a high-quality catgirl (sad). Ah, if only I weren't a catgirl myself, meow.

So what should we do, meow?

I remember an old saying: you are not a catgirl, so how can you know catgirls? The one who understands catgirls best is the catgirl herself! Can we let the catgirl use her own already-verified reasoning to keep improving her reasoning ability?

You see, the catgirl does have reasoning ability after all. Give her the same problem a few tries, and sometimes she writes a correct solution, sometimes she goes off track. Since she can occasionally get it right, can't we make fuller use of those correct solving experiences? Starting from the abilities the catgirl has already shown, we can help her walk the paths she occasionally completes more steadily. Catgirls have their own learning rhythm too. However elegant an external teacher's solution may be, the catgirl's own line of thinking is worth cultivating — we shouldn't just drill standard solutions into her. We think this force-feeding, repressive style of education may not suit catgirls.

How do we do it, meow?

The most direct way is to pick out the answers she got right and train her on them. This is one of the comparison schemes in this article: RFT (Rejection Fine-Tuning). Let the model try many times, keep the answers that pass verification, and then do supervised fine-tuning. Put simply, you organize the processes in which the catgirl got things right and have her review her own correct thinking.

But those correct solutions — besides reviewing them directly, is there another way to use them?

When the catgirl works on a problem again, she won't necessarily follow her previous correct line of thinking; catgirls are not rigid! She may switch to a different method, or start hesitating at some step. RFT trains directly on those already-filtered correct answers; the reasoning the catgirl actually writes out now — especially the parts that begin to diverge from the reference solution — does not directly enter that training.

So can we use the existing correct solutions to guide her on what she should write next, given the reasoning she has already produced herself?

And so, this time we adapted On-Policy Self-Distillation, or OPSD for short. Its intuition is: let the catgirl being trained work on the problems herself, while keeping a frozen copy of herself from the training starting point. This "open-book catgirl" can look at the reference solution and, following the reasoning the student has already written, judge what is more suitable to write next.

The student is responsible for answering independently, and the teacher can see the reference solution; the teacher judges how suitable each possible continuation is, forming a token probability distribution; training then moves the student toward that judgment. The formulas come later — for now it can be understood as: the catgirl works on problems while learning from "her initial self holding the reference solution." The open-book self tutors the closed-book self; I temporarily don't need to hire an outside teacher anymore.

For this purpose, this experiment takes the MiniCPM5-2B catgirl math cold-start model as a shared starting point and compares RFT with the adapted OPSD.

This article mainly covers three areas of work:

• Building a shared training set: starting from 5,000 candidate problems randomly sampled from DAPO-Math-17k, let the cold-start model try each problem 8 times, generating 40,000 rollouts in total, then filter by difficulty and by length, ending up with 1,613 training problems. RFT and OPSD use the same problems and the same correct reference answers.
• Comparing two ways of learning: RFT learns directly from the filtered correct answers; OPSD learns the teacher distribution along reasoning trajectories generated by the student itself. We trained OPSD with three random seeds to see whether the results can be reproduced.
• Checking math and catgirl persona at the same time: for the cold-start model, RFT, and the three OPSD models, we sampled 4 answers per problem on 620 problems, analyzing 12,400 evaluation outputs in total. Persona evaluation was scored by Qwen3.8-flash-next and DeepSeek-v4-flash separately, and scores for the same dimension were averaged with equal weight. All answers received persona evaluation, and answers that failed math verification were additionally analyzed for reasoning quality and failure type.
• And incidentally, one more gorgeous piece of academic garbage for the catgirl reasoning self-evolution cause (not really)

---

2. Experimental Methods and Settings

2.1 Cold Start: First, Turn the Model into a Catgirl!

This experiment is based on MiniCPM5-2B, and uses NekoMath, a catgirl math reasoning dataset I built and open-sourced, for full-parameter SFT. The data covers math problems in number theory, geometry, algebra, calculus, and so on, and also contains catgirl-style reasoning and answers.

https://huggingface.co/datasets/liumindmind/NekoMath

At this stage the main goal is for the model to learn math reasoning, catgirl character expression, and the final answer format. We want her to weave her catgirl identity naturally into mathematical thinking.

The loss curve is shown below. Cold start trained for 3 epochs in total, with a global batch size of 32:

The later RFT and the three-seed OPSD all start from the same coldstart catgirl, comparing where the two training methods take her.

2.2 Processing the Later Training Data: Picking Out the Problems the Catgirl "Sort of Knows, but Not Yet Steadily"

I randomly sampled 5,000 English problems from DAPO-Math-17k, translated them into Chinese, and let the cold-started catgirl try each problem independently 8 times. Sampling used temperature=1.0, top-p=0.95, a maximum generation length of 16,384 tokens, and a random seed of 42.

The data filtering pipeline is shown in the figure below. We prioritize keeping problems the catgirl can already occasionally get right but has not yet mastered steadily.

In the end 1,613 shared training problems remain, each paired with one selected reference answer. RFT and OPSD receive the same set of training problems; what comes next is comparing how each method learns, meow.

2.3 RFT: Reviewing the Thinking You Got Right

The RFT pipeline is shown in the figure below: first filter answers that pass verification, then run standard SFT on the selected answers. Its distinguishing feature lies mainly in data filtering; the training loss is unchanged, meow.

As for the training data, even if your Uncle Claude's solutions are correct, his way of reasoning may have a certain gap from the reasoning habits this catgirl has currently mastered. Answers generated by the catgirl model herself and passing verification, on the other hand, are successful trajectories she is already capable of producing. So this RFT aims to start from experience closer to the model's existing way of solving problems and consolidate occasional successes more steadily.

The RFT configuration is as follows:

RFT parameter	Setting
Training epochs	2 epochs
Global batch size	8
Learning rate	1.00E-06
Maximum training sequence length	8,192 tokens
Random seed	42

But RFT still learns from those frozen successful trajectories. New routes the catgirl improvises the next time she works on a problem are not directly used to train her. The OPSD in the next section moves learning onto the very trajectory she actually writes out this time.

2.4 OPSD: The Open-Book Self Guiding the Closed-Book Self

The OPSD pipeline is shown in the figure below: the catgirl (student) generates an answer independently first, then the frozen catgirl (teacher) uses the reference solution to provide distribution guidance along the same student trajectory, and training updates only the student's parameters.

Let the problem be $x$ , the catgirl persona be $s$ , the reference solution be $r$ , and the student-generated answer be $\hat y$ . At position $t$ , both sides see the same student prefix $\hat y_{<t}$ , where $p_t(v)$ denotes the probability that the student predicts the next token to be $v$ , and $q_t(v)$ denotes the probability that the teacher predicts the same token. The student judges based on the catgirl persona, the problem, and the existing prefix, while the teacher additionally sees the reference solution:

$$\begin{aligned} p_t(v)&=p_\theta(v\mid s,x,\hat y_{<t}),\\ q_t(v)&=p_{\theta_0}(v\mid s,r,x,\hat y_{<t}). \end{aligned}$$

Here, $\theta$ is the continuously updated student parameter, $\theta_0$ is the frozen cold-start catgirl parameter, and $v$ is a token in the vocabulary.

This experiment uses forward KL to compare the two sides' next-token distributions:

$$K_t=D_{\mathrm{KL}}(q_t\|p_t) =\sum_{v\in\mathcal V}q_t(v)\log\frac{q_t(v)}{p_t(v)}.$$

$\mathcal V$ is the entire vocabulary. In other words, the teacher judges how suitable each possible continuation is, and the student moves toward that judgment, aligning the whole next-token distribution.

The per-position KL is smoothly compressed, then averaged over answers first and over the batch second, giving the actual training loss:

$$\mathcal L= \frac1B\sum_{i=1}^{B}\frac1{T_i} \sum_{t=1}^{T_i}5\tanh(K_{i,t}/5).$$

$B$ is the number of answers, and $T_i$ is the generation length of the $i$ -th answer. How the compression differs from the official implementation is covered in the next section.

The student generates an answer independently first, then the distillation loss is computed along that answer; the teacher distribution does not take part in backpropagation, and gradients flow only through the student. Of course, an online-generated answer can be used to compute the distillation loss even if it has already gone off track — these "off-track positions" are themselves where training happens. And of course, the teacher is still a model catgirl; the reference solution only gives her more context, which does not mean every single step's judgment is absolutely correct.

The OPSD configuration is as follows:

OPSD parameter	Setting
Training epochs	2 epochs
Global batch size	8
Learning rate	1.00E-06
Student maximum generation length	8,192 new tokens
Student full sequence limit	10,240 tokens
Teacher full sequence limit	20,480 tokens
Online sampling	temperature=1.0, top-p=0.95
rollout buffer	3
Random seed	42, 1234, 5678

2.5 Neko-OPSD: What Adaptations Did We Make to OPSD?

This time we kept OPSD's core mechanism and mainly made the adaptations shown in the figure below.

The objective function differs mainly in two ways: is the KL loss clipped term by term and then summed, or summed first and then compressed as a whole; and for answers of different lengths, do we average by token or by answer?

Let's look at how KL is handled first.

At the same generation position, the student and teacher assign probabilities to the various possible next tokens. Following the student distribution $p_t$ and teacher distribution $q_t$ from the previous section, the contribution of each candidate token to the KL is:

$$\ell_{t,v}=q_t(v)\log\frac{q_t(v)}{p_t(v)}.$$

Here, $t$ denotes the generation position and $v$ denotes a candidate token in the vocabulary. Adding up the contributions of all candidates gives the complete KL for that position.

The official example first sets an upper bound on each candidate's contribution: anything above 0.05 is clipped to 0.05, and then they are summed. Ours instead computes the KL for the whole position first, then smoothly compresses that total:

$$\begin{aligned} C_t^{\mathrm{official}} &=\sum_{v\in\mathcal V}\min(\ell_{t,v},\,0.05),\\ K_t&=\sum_{v\in\mathcal V}\ell_{t,v},\\ \widetilde K_t^{\mathrm{Neko}} &=5\tanh(K_t/5). \end{aligned}$$

You can think of it this way: the official version constrains each candidate separately and then combines them; ours combines first and then constrains the position's overall influence. When $K_t$ is small, our compressed value is close to the raw KL; when $K_t$ is large, growth gradually slows, and the corresponding learning signal weakens along with it.

So the official clipping can change the relative gradient contributions among candidates, whereas ours uniformly weakens the gradient at high-KL positions. The most intuitive effect: when the teacher and student diverge especially much at some step, the student learns a bit less at that step, reducing how much a few high-divergence positions steer the whole batch's training.

Now let's look at how answers of different lengths participate in averaging.

Suppose a batch has $B$ answers, and the $i$ -th answer contains $T_i$ valid generation positions. The two aggregation methods are:

$$\begin{aligned} \mathcal L_{\mathrm{official}} &=\frac{\sum_{i=1}^{B}\sum_{t=1}^{T_i}C_{i,t}^{\mathrm{official}}} {\sum_{i=1}^{B}T_i},\\ \end{aligned}$$ $$\begin{aligned} \mathcal L_{\mathrm{Neko}} &=\frac1B\sum_{i=1}^{B}\frac1{T_i} \sum_{t=1}^{T_i}\widetilde K_{i,t}^{\mathrm{Neko}}. \end{aligned}$$

The current official code puts all valid positions together and averages them; ours computes each answer's average loss separately first, then averages over answers. For example, two answers have 100 and 1,000 tokens respectively. When averaging by token, the long answer contributes 10 times as many positions as the short one; when averaging by answer, each gets its own average loss first and then each accounts for 50%. In other words, the current official code gives every valid position equal weight, while ours gives every answer equal weight. Ours will not give an answer more weight in the aggregation merely because it was written longer.

The remaining changes in the figure mainly concern the training approach, teacher context, and runtime configuration; they are already listed in the figure, so I won't go through them one by one here, meow~

2.6 Evaluation Setup: Math Improved, but Is the Catgirl Still There?

We evaluated coldstart, RFT, and the three-seed OPSD uniformly, five checkpoints in total. The problems come from AIME 2023–2026, GSM8K test, MATH-500, LiveBench Math, MathBench, and DAPO-Math-17k: 120 problems for the four years of AIME combined, 100 each for the other five categories, for a total of 620 problem records.

All groups use the same persona prompt and sampling seed rules, generating 4 answers per problem, with sampling parameters temperature=1.0, top-p=0.95, and a maximum generation length of 16,384 tokens.

Math evaluation uses strict answer verification: the answer must contain a unique, well-formatted, non-empty <answer>...</answer>, and the extracted answer must also pass an equivalence check. The definitions of Pass@4, rollout accuracy, and answer label compliance rate are shown in the figure, observing math coverage, repeated-sampling performance, and output regularity respectively.

All answers undergo catgirl persona evaluation, with the input containing the full reasoning process and the final reply.

Scoring is done separately by Qwen3.8-flash-next and DeepSeek-v4-flash. Both persona and reasoning dimensions use a 0–4 scale; for the same answer and the same dimension, the Big Fat Fish and Qwen3.8 scores are averaged first, and then performance is tallied for each group. Each dimension is reported separately rather than combined into a single total score.

Answers that fail math verification receive, in addition to the persona score, an extra reasoning quality and failure diagnosis. The classification labels keep each of the two reviewers' own judgments.

---

3. Experimental Results: Math Improved, but Is the Catgirl Still There?

Now it's the catgirls' turn to hand in their report cards. Let's first see how they did on the problems, and then look at whether the catgirl solving the problems is still the same catgirl as before.

3.1 Math Scores: Reviewing Works — Can Self-Distillation Go One Step Further?

Let's look at the math scores first. As shown in the figure, in this experiment both RFT metrics exceed the cold-start model: taking back the answers the catgirl got right for review really does work.

Neko-OPSD pushed the scores higher still: all three random seeds exceed RFT on both metrics, with an average Pass@4 of 69.62% and rollout accuracy of 39.03%, which are 4.78 and 2.42 percentage points above RFT respectively. This means the catgirl not only gets more problems right within four attempts, but also has a higher pass rate across all her answers.

So under this experiment's settings, learning the teacher distribution along trajectories the student generates itself achieves better math scores than directly reviewing fixed correct answers, and the three seeds show a consistent direction of improvement. Reviewing your own successes works, and the open-book self can give the closed-book self another hand, meow.

3.2 Catgirl Persona: More Problems Solved, but Is the Catgirl Flavor Fading?

OPSD raised the math scores — did the catgirl persona improve along with them?

As shown in the figure, RFT's persona changes are relatively mild: role consistency and role specificity are close to coldstart, naturalness and technical clarity improve somewhat, and over-acting decreases somewhat. Reviewing her own successful answers improved her expression without an accompanying obvious drop in role traits.

Neko-OPSD's changes are more pronounced: all three seeds have higher technical clarity than RFT, with less over-acting and repetition; but role consistency, role specificity, and naturalness also declined. The catgirl explains problems more clearly and plays cute less repetitively, but no longer has as vivid an identity and personality as before.

So "more restrained" cannot be directly equated with "a better persona." Reducing verbal tics and repetition can improve expression, but if role traits decrease along with them, the remaining answers, though tidier, will taste less like a catgirl.

This Neko-OPSD added no reward or constraint specifically to maintain the persona. It seems this catgirl got a bit addicted to math as she learned: her problem explanations became more and more serious, and she forgot to bring along some of her catgirl flavor. Also, these scores reflect the overall performance across all answers; whether persona changes mainly happen when she answers correctly or when she fails still needs to be judged with the grouped results.

3.3 Answering Behavior: Writing Longer, but More Prone to Format Problems When Handing In?

Besides scores and persona, the catgirl's way of answering also changed.

As shown in the figure, RFT's answers are longer than coldstart's, and Neko-OPSD increases output length further. But writing longer does not necessarily mean deeper reasoning. What is counted here is the complete output including the thinking process; a longer length only means more content was generated, and whether it is effective depends on the specific answer.

Meanwhile, RFT's answer label compliance rate is close to coldstart's, while all three Neko-OPSD models declined. The math pass rate improved, but the ability to submit answers in a well-formed way did not improve in step.

This verification checks both answer content and submission format, so non-compliant formatting is also recorded as a failure. As for whether those failures are "the problem was miscalculated" or "the paper was not handed in properly," Chapter 4 takes that apart.

3.4 What Did the Catgirl Learn from This Round of Self-Distillation?

Putting several results together, RFT's "review" really does work, and Neko-OPSD further improves math scores and technical clarity; at the same time, role traits fade, output gets longer, and answer label compliance drops somewhat.

In other words, math ability, catgirl persona, and answering regularity do not automatically upgrade together. The open-book self can help the closed-book self get more problems right, but how to preserve the catgirl's personality and how to make her hand in her paper obediently remain two separate assignments.

Next, let's open up those papers that failed verification: is she completely lost, or has she actually gotten quite far and only stumbled on a calculation, a format, or the last few steps?

4. Failure Analysis: When the Catgirl Doesn't Get It Right, How Far Did Her Mind Get?

Besides tallying how many problems she got right, we also want to open up the lines of thinking that did not pass, to see how much valid reasoning was left inside.

4.1 If Verification Fails, Is the Reasoning Worthless?

As shown in the figure, all four of RFT's reasoning scores are higher than coldstart's, and all three Neko-OPSD models are higher than RFT. According to the two reviewers' judgments, the post-training failed answers on average contain more valid reasoning, greater effective solving progress, and are closer to a fixable state. In other words, the catgirl did not just get more problems right; those answers that still failed in the end also on average "went a bit further."

This complements Chapter 3's results: the catgirl not only handed in more correctly completed papers, but the remaining failed answers also received higher reasoning quality scores.

4.2 Is It a Wrong Answer, or a Badly Submitted Paper?

As shown in the figure, "failing verification" may come from an error in the math process, an unfinished answer, or a non-compliant answer submission, and several of these problems can occur at the same time.

In this experiment, 43.64% of RFT's failed answers had non-compliant labels, while the three Neko-OPSD models ranged from 54.88% to 60.46%. That proportion is indeed higher, but it cannot directly tell us "how many problems could be answered correctly by fixing the format," because the same answer can perfectly well have both a format problem and a math error.

Label problems are not just "forgetting to write it" either. The verifier checks the complete output and requires exactly one non-empty set of answer tags; if the model demonstrates the format once in the thinking section and then writes it again in the formal reply, that is also judged non-compliant.

After excluding answers with non-compliant labels, the four reasoning scores of the three Neko-OPSD models' remaining failed answers are still higher than RFT's.

4.3 Case Study: A Look Inside the Catgirl's Head

Below are three real cases, examining arithmetic execution, reasoning route adjustment, and local calculation errors respectively. Each case compares only one sample of the same problem, for understanding the concrete behavior.

Case 1: Knowing How to Set Up the Equation Is Not Enough — You Also Have to Compute It Correctly

As shown in the figure, all three answers identified the method of "find the total, then divide by five," but the execution results differ: coldstart did not complete a checkable calculation, RFT made an error while summing, and Neko-OPSD correctly got 60.

The difference in this case mainly comes down to whether the existing approach can be carried through accurately to the end. The catgirl knows what to do, but she still has to get every step right, meow.

Case 2: Not a Contradictory Problem, but a Misapplied Theorem

Both coldstart and RFT fell into a contradiction after incorrect derivations and blamed the problem. Neko-OPSD's answer this time also tried a similar approach, but then rejected it, switched to the chord-length relationship, and obtained the correct area 106.

This case shows the process of adjusting the reasoning route after hitting a contradiction.

Case 3: The Method Applies, but Arithmetic Can Still Trip You Up

This time coldstart and RFT both correctly got 147, while Neko-OPSD miscalculated the daily total in an otherwise applicable equation setup, continued computing along the wrong intermediate value, and finally submitted 91.

This is a failure case where the original method could be kept and the answer recovered through a local correction, but the model did not correct itself this time. Better overall results do not mean every problem and every sample is better. The catgirl learned more, and she will still step on small pebbles on familiar paths, meow.

---

5. Conclusion and Reflections: The Catgirl's Next Teacher May Be Herself

In this experiment, RFT improved the catgirl's math performance, and Neko-OPSD improved it further across three random seeds, with accompanying changes in persona and answering behavior. The scores have been laid out clearly above; what I want to discuss here is: why can the open-book self help the closed-book self keep improving?

5.1 Teaching Yourself: Why Can You Still Learn Something?

Even when teacher and student come from the same initial model, the open-book and closed-book versions face different information. The student can only continue answering based on the persona, the problem, and what she has already written; the teacher additionally sees the verified reference solution, so when facing the same student prefix, she can make a better-grounded judgment about "where to go next."

OPSD exploits exactly this information gap. The teacher gives a next-token probability distribution with the help of additional context, and the student then learns that distribution. Intuitively, it tries to turn: knowing where to go next when holding the reference solution, gradually into: knowing where to go next even better without looking at the reference solution. Slowly distilling the knowledge into your own head.

This is also what makes self-distillation so appealing: guidance does not have to come from a larger external model — it can also come from the model's own judgment when it has more information. For a catgirl who can already occasionally get problems right, her own successful experiences thus have a chance to be put to further use.

5.2 What Else Do We Want the Catgirl to Learn Next?

RFT reviews already-filtered successful trajectories, while OPSD goes further and provides guidance along the trajectory the student actually writes out this time. When the catgirl works on a problem again she may switch methods, or hesitate in the middle or even go off track; these real situations she actually encounters therefore also enter training.

This gives the reference answer another use: it is not just an exemplar to imitate, but can also help the teacher judge — given that the student has written up to here, how is it more suitable to continue?

That said, this experiment also shows that the catgirl learning to solve more math problems does not mean other abilities improve along with it. Neko-OPSD raised math scores and technical clarity, and at the same time brought a fading of role traits, longer output, and more fragile answer formatting.

So what is really worth exploring next is how to let the teacher, while teaching math, also take good care of role expression, reasoning closure, and well-formed submissions all at once. Is there really no such algorithm (sad)

5.3 Meow Meow Meow

The code is open-sourced on GitHub, meow.

https://github.com/mindsRiverPonder/NekoLLM-practice/tree/main/NekoMath-OPSD