Don't Stop LCCP: A Scaling Law for Long-Context Continual Pre-Training
Authors: Ziqing Qiao, Xiaoyue Xu, Sikui Zhang, Yinuo Xu, Bingxiang He, Chaojun Xiao, Xu Han
✨ The first scaling law for long-context performance under the "short-then-long" pre-training paradigm: model scale × short-context data × long-context data, all three variables modeled at once, with extrapolation error under 2%.
✨ The Constant-Last4 protocol: a training trajectory you can extend and observe stably, without changing the budget and retraining from scratch.
✨ Gains that persist and carry over: the extra budget spent at 32K keeps paying off through 128K extension, long-context SFT and agentic tasks.
The project is ongoing; the paper, code and models are all publicly available.
In recent years, the common recipe for long-context models has been: finish short-context pre-training, then add an "extension" stage (long-context continual pre-training, LCCP), continuing training on long-sequence data for a short stretch — perhaps a few billion or tens of billions of tokens. This stage is usually brief. It is widely assumed that a limited amount of long-context data is enough to train long-context ability fairly well, and that there is no need to keep scaling it the way we scale short-context pre-training. But with long-horizon tasks mattering more than ever, it may be worth asking: if we keep going, does long-context performance keep improving? Do we often call it quits at the halfway point? We ran a systematic experiment across 5 model scales and 100+ training configurations. We successfully fit a scaling law for long-context performance under the short-then-long pre-training paradigm, monitored a 100B-token LCCP trajectory on a 1B-scale model and the gains it kept producing, and verified on a wide range of downstream long-context and agentic tasks that these gains persist. The core finding: don't stop LCCP too early — the gains are visible, predictable and portable.
The paper, code and models in this article are all open-sourced.
Why Focus on Training for Long-Context Ability?
Long-context ability is one of the foundational capabilities of large models, underpinning complex long-range tasks and agentic workflows. But training on long sequences from the very start of pre-training is expensive. So the mainstream approach today is: finish short-context pre-training (say, 4K), then run continual pre-training (LCCP) on long-sequence data (32K / 128K) to stretch the context window and improve long-context performance. Many works extend the context window with a "relatively limited training budget"; some models spend trillions of tokens on long-context training. But how long-context performance improves as the number of LCCP training tokens grows has not been well characterized. So we ask: does continuing to scale LCCP deliver continued long-context improvement? And can those gains be predicted in advance?
A Scaling Law for Long-Context Performance That Models Three Variables Jointly
Unlike pre-training from scratch, LCCP starts from a model that has already learned basic abilities — which means its long-context performance may depend simultaneously on model scale, how many tokens went into short-context pre-training, and how many tokens go into long-context training. We model all three at once: non-embedding parameter count N, short-context pre-training tokens Ds, and long-context training tokens Dl. Using LongPPL as a proxy for long-context performance, we fit and extrapolate across 5 model scales and 109 training configurations, and find that an additive power law with an offset term accurately captures the observed trends.
We ran two extrapolation experiments: fit on S1–S3 (31M–104M non-embedding parameters) and extrapolate to S4 (245M non-embedding parameters); and fit on S1–S4 and extrapolate to S5 (821M non-embedding / 1.22B total parameters). Results: S1–S3 → S4 extrapolation gives R²=0.97 and MAPE=1.4%; S1–S4 → S5 gives R²=0.95 and MAPE=1.7%. This compact scaling law successfully fits long-context performance across model scales and short- and long-context data volumes; we understand it as a scaling law valid within a bounded range. Its significance: if you know the model scale, how long short-context pre-training ran, and how much budget you plan to give long-context training, it can tell you in advance roughly what long-context performance to expect.
Constant-Last4: A Protocol You Can Extend and Observe Stably
With the scaling law in hand, the next question is: how do we keep LCCP running? If we want to observe long-context performance continuously along a real, ever-extending training trajectory, how should we do it?
Conventional LCCP either uses a learning rate with a decay schedule — stable and reliable, but the LCCP training budget must be fixed in advance, which makes flexible continuation inconvenient — or uses a constant learning rate, which supports continuation at any time but, in our measurements, fluctuates more than the decay schedule and yields significantly higher PPL across metrics.
We therefore introduce the Constant-Last4 protocol: train with a constant learning rate, and when observing, average the weights of the most recent 4 consecutive checkpoints (which we call Last4). Results show that under this training-and-observation protocol the fluctuation disappears, and PPL matches the decay schedule almost exactly.
Constant-Last4 gives us a reliable, extendable training trajectory. On S5, starting from Ds=450N, we ran with a constant learning rate all the way to Dl=120N (about 100B long-context tokens), saving checkpoints periodically and observing them with Last4. This trajectory's training data volume reaches more than 8× the largest budget in the fitting set, yet the scaling law's predictions remain accurate. Going further: if we fit using only the first 40N of the trajectory, can we predict performance at 120N? Our experiments say yes. That means you can work out where the finish line is before running the whole race.
Gains That Persist and Carry Over
LongPPL is, after all, only a proxy for long-context performance. When LongPPL drops, does downstream long-context performance actually improve end to end? For different long-context training budgets, we tested performance on 32K RULER, and also — after 128K context extension, long-context SFT and agentic SFT — verified on a range of mainstream long-context and agent benchmarks how model capability changes with different amounts of LCCP training.
RULER-32K evaluation shows Constant-Last4's RULER scores rising steadily, with no sign of saturation within the evaluated range. After 128K extension: the larger the 32K training budget, the better the post-extension performance — the extra tokens spent at 32K carry over to 128K. After SFT: downstream performance improves as the LCCP budget grows. We observed this trend not only after long-context SFT on long-context downstream tasks. After agentic SFT, we also observed sustained model improvement on tool-use and search-related agent tasks such as tau²-telecom, GAIA and BFCL-WebSearch.
Closing Thoughts
Don't Stop LCCP is not meant to prove that "long-context training always pays off." It is a reminder: before declaring "that's enough," look at where the curve actually is — the finish line you imagine may be just a halfway aid station. In other words, we have installed a "dashboard" for long-context continual pre-training: it can predict before you run, and let you watch while you run, so you don't have to restart with a new budget every time. If you are training long-context models, give it a try: run LCCP to its full extent, observe it stably with Constant-Last4, predict ahead with the scaling law — and see how far it can take you.