Research Highlight | Can Linear Attention "Steal the Job" Too? Let Full-Attention Layers Learn First for More Accurate Long-Range Retrieval
Authors: Xingyu Shen, Yingfa Chen, Ziqing Qiao, Chaojun Xiao, Xu Han, Zhiyuan Liu
✨ First to identify the "Competing Learning" phenomenon: full-attention layers and RNN layers compete to learn the same long-range retrieval ability, and whichever learns it first suppresses the learning signal for the others.
✨ Validated in large-model training: we propose the BLD method, which freezes RNN layers during the first 10% of training to encourage the right division of labor. Both language modeling and retrieval tasks improve substantially.
✨ Inside the model: tracing attention heads within the model, we find that models trained with BLD's correct division of labor have noticeably more retrieval heads.
The paper and code are both open-sourced.
Full-attention layers are naturally better at long-range retrieval, yet they may be beaten to it by RNN layers during training. A Tsinghua University research team first observed this competition on a synthetic retrieval task, then applied the training method it inspired to language models of up to 1B parameters. By adjusting only when RNN layers start updating, hybrid models at all six scales achieved better long-range retrieval scores.
Background
More and more large language models combine full-attention layers with RNN layers to form hybrid architectures. Qwen3.5-397B-A17B uses Gated DeltaNet plus full attention; Kimi K3 uses KDA (the representative RNN layer used in Kimi K3) plus Gated MLA; NVIDIA Nemotron 3.5 Super uses Mamba plus full attention.
This design rests on an assumed division of labor: RNN layers handle short-range token mixing, while full-attention layers handle long-range retrieval — for example, recovering a name or a verbatim phrase from thousands of tokens earlier. Prior work has observed this functional division after training. But during pre-training from scratch, do the two kinds of layers naturally form it?
We used a synthetic retrieval task to watch the training process. The model has to skip over 64 or 8,192 blank tokens and repeat a string of random digits from the beginning. By substituting the output of a given layer at the answer position, we can estimate that layer's contribution to recovering the correct digits.
Direct Training | On a Synthetic Retrieval Task, the RNN Layer Gets There First
When the two kinds of layers train together, the full-attention layer initially learns some retrieval ability. But as the KDA layer catches up and overtakes it, the full-attention layer's contribution to retrieval quickly drops to near zero. In this synthetic task, the retrieval work intended for the full-attention layer ends up mainly handled by the RNN layer.
Figure 1 | Training curves on the synthetic retrieval task. Red is the full-attention layer, green the KDA layer. The vertical axis measures each layer type's contribution to the retrieval answer. Adapted from Figure 2 of the paper.
We call this phenomenon Competing Learning. If we pause updates to the RNN layer mid-training, can the full-attention layer learn the ability back?
Freeze Experiment | With RNN Layers Paused, Can Attention Catch Up?
We reused the same synthetic retrieval task, freezing KDA's parameters at different training steps. During freezing, KDA still participates in the model's computation while everything else trains as usual. With joint training throughout, the full-attention layer's retrieval contribution briefly rises and then falls to near zero. When KDA is frozen in time, the full-attention layer's retrieval contribution keeps rising instead.
More interesting is what happens after unfreezing. We resumed updates for all KDA variants frozen at different times at step 500. In the earlier-freezing groups, the full-attention layer had already built a retrieval advantage, and its contribution kept rising after joint training resumed. In some of the later-freezing groups, the full-attention layer's contribution fell again as soon as KDA resumed updating. Unfrozen at the same step 500, whether the attention layer can hold on to retrieval depends on how much it had already learned by then.
Figure 2 | Freeze and unfreeze experiments on the synthetic retrieval task. The left plot compares different freezing points; the right plot resumes KDA updates for all groups at step 500. Blue-purple curves correspond to earlier freezing; the dashed line is the joint-training baseline throughout. Adapted from Figure 4 of the paper.
Gradients changed too. Compared with the baseline at the same step, the full-attention layer's gradients grew steadily once KDA was frozen. After KDA updates resumed, this amplification weakened markedly. This is consistent with our explanation of Competing Learning: how fast the RNN layer learns affects how much learning signal the full-attention layer still receives.
In the same synthetic setup, we also varied the ratio of long- and short-distance questions. The more long-distance questions, the larger the full-attention layer's retrieval contribution. Short-range patterns are more common in natural language, and KDA more easily exploits nearby information early in training. This offers an explanation for why RNN layers tend to get there first — and raises the next question: when pre-training a language model on real text, can we deliberately change this division of labor?
Large-Model Validation | Pre-Training on Real Corpora, Letting Full-Attention Layers Train First
The first two experiments observed the training mechanism on a synthetic retrieval task. Next, we pre-trained KDA hybrid language models from scratch at six scales, from 180M to 1B parameters, on the real-text corpus DCLM, to test whether this training intervention improves language-model performance.
We call this training method Better Layer Division (BLD). For roughly the first 10% of training tokens, the RNN blocks stay frozen while the full-attention blocks, word embeddings and final normalization layers keep updating. The RNN blocks are then unfrozen, keeping their existing weights, and all layers finish the remaining training together. At each scale, the two model groups use the same architecture, data, token budget and optimizer schedule.
Language-modeling loss dropped at all six scales, and average scores on Recall and the long-context benchmark RULER both rose. For the 1B model, for example, RULER improved from 32.75 to 36.04, Recall from 36.50 to 38.14, and long-context perplexity LongPPL fell from 7.69 to 6.50. Common-sense reasoning changed less, with four of the six scales improving. Replacing KDA with Gated DeltaNet or Gated Linear Attention also brought retrieval gains to a model of about 400M parameters.
Figure 3 | Zero-shot evaluation results after pre-training on the DCLM corpus. RULER aggregates 57 task-length combinations over the 1K–16K token range. Adapted from Table 1 of the paper.
Division of Labor Inside the Model | From One Key Retrieval Head to Three
We also examined a 510M-parameter model fully pre-trained on real corpora. In a long-document retrieval test, we ranked attention heads by how much they depend on the full context, from low to high, then progressively restricted the context they could see and observed when retrieval accuracy dropped.
In the conventionally pre-trained model, retrieval accuracy dropped markedly mainly when the last key KV head was restricted. In the BLD model, it dropped markedly as the last three key KV heads were progressively restricted. This indicates that, in the language models tested, BLD training led more attention heads to participate in long-range retrieval.
Figure 4 | Attention-head analysis of the 510M-parameter model. Left: conventional pre-training; right: BLD. Brighter color means the head group relies more on the full context. The four query heads in a group share one KV head. Adapted from Figure 9 of the paper.
Closing | Whoever Learns First Is More Likely to Own Retrieval
The two kinds of layers in a hybrid architecture compete to learn the same ability. Whichever layer gains the retrieval advantage first lowers the model's prediction error on the task sooner, weakening the learning signal available to the others and thereby suppressing their retrieval learning. In the synthetic retrieval task, the RNN layer usually gets the head start. But if the RNN layer is frozen early enough for the full-attention layer to establish a retrieval advantage first, that division of labor can persist after joint training resumes.
We applied this finding to pre-training on real corpora. By delaying when RNN blocks start updating, BLD gave all six hybrid models from 180M to 1B parameters better retrieval scores and lower language-modeling loss. In the 510M model, attention-head analysis further found that the key KV heads responsible for long-range retrieval increased from one to three.
Of course, these conclusions have boundaries: evaluation covers six scales from 180M to 1B parameters, and behavior at larger scales remains to be verified; the phenomenon was mainly observed on the synthetic retrieval task and the DCLM corpus, and whether it holds under other data distributions needs more experiments; BLD uses a fixed freezing ratio and duration, and the trade-off between cost and benefit is left for future exploration.
Want to verify it yourself? Clone the code repository and run BLD on your own hybrid architecture; we would also love to hear about any "job-stealing" cases you observe during training — the next question might come from your experiments.
In one sentence: whoever learns first is more likely to own retrieval, so let the right layer learn first.
✍️ Authors: Xingyu Shen, Yingfa Chen, Ziqing Qiao, Chaojun Xiao, Xu Han, Zhiyuan Liu
Original article link: https://arxiv.org/abs/2610.26830