Innovating Transformer: a 9B model on the sparse-linear hybrid architecture SALA runs million-token context on-device

2026-02-11

As is well known, Transformer and its core Full Attention mechanism have long held the central place in large-model architectures, but quadratic computational complexity and high VRAM usage have long been the roadblocks to ultra-long context processing and large-scale model applications. Daring to challenge this established authority requires not only the courage to innovate toward the long-term goal of AGI, but also a distinctive technical vision and the hard strength to break through technical barriers. From DeepSeek's sparse attention (DSA) and MiniMax's linear attention to Moonshot AI's linear attention (KDA), everyone is entering the arena of attention architecture innovation.

Today, OpenBMB takes a key step in this transformation: we officially release SALA, the industry's first sparse-linear hybrid attention architecture trained at scale (Sparse Attention-Linear Attention, SALA for short), along with the text model MiniCPM-SALA built on it, pursuing longer text processing capability and ultimate inference cost-effectiveness. MiniCPM-SALA not only shows clear advantages in multiple long-text understanding and generation evaluations, but also maintains excellent performance comparable to same-scale full-attention models in core capability dimensions such as knowledge Q&A, mathematical reasoning, and code generation, achieving comprehensive performance that is "strong at both long and short" and ultimate inference cost-effectiveness.

➤ MiniCPM-SALA at a glance

Architecture innovation: a sparse-linear hybrid attention architecture that significantly reduces inference overhead and VRAM usage while overcoming the accuracy bottleneck of pure linear architectures in long-range information recall, balancing efficiency and performance.

Outstanding performance: HyPE (hybrid position encoding) effectively reconciles performance on short and long contexts. While matching other 8B full-attention models on general capabilities (such as knowledge, math, and coding), MiniCPM-SALA shows clear advantages on multiple long-context benchmarks.

Efficient inference: without acceleration algorithms such as speculative sampling, on cloud inference chips it runs up to 3.5× faster than Qwen3-8B at a sequence length of 256K tokens, and supports inference with context lengths up to one million tokens on both cloud chips and consumer on-device GPUs.

➤ Related links

GitHub:

🔗 https://github.com/openbmb/minicpm

Hugging Face:

🔗 https://huggingface.co/openbmb/MiniCPM-SALA

ModelScope:

🔗 https://www.modelscope.cn/models/OpenBMB/MiniCPM-SALA

GitCode:

🔗 https://ai.gitcode.com/OpenBMB/MiniCPM-SALA

MiniCPM-SALA technical report:

🔗 https://github.com/OpenBMB/MiniCPM/blob/main/docs/MiniCPM_SALA.pdf

Why a hybrid architecture is the better answer

Traditional Transformer models processing long text suffer from full attention's quadratically growing computation with text length, while the key-value pairs cached during generation also consume large amounts of VRAM. Existing solutions fall into two main schools, linear attention and sparse attention, but pure linear attention has the shortcoming of accuracy loss, and pure sparse attention is limited by a storage bottleneck.

MiniCPM-SALA pioneers a sparse-linear hybrid attention architecture that neatly resolves this core tension. 75% of the model's layers use linear attention (Lightning Attention), responsible for precisely capturing key local information. The remaining 25% use sparse attention (InfLLM-v2), focused on efficient global information flow. This golden ratio balances computational efficiency and semantic accuracy.

MiniCPM-SALA model architecture

The related research papers are now public:

Sparse-linear hybrid attention: https://arxiv.org/pdf/2601.22156

InfLLM v2: https://arxiv.org/pdf/2509.24663

Not just long, but strong

MiniCPM-SALA has excellent length generalization. Without any additional techniques (such as YaRN), it can extend context length to 2048K.

Evaluating efficiency at sequence lengths from 64K to 1024K tokens on cloud inference chips, MiniCPM-SALA shows a significant efficiency advantage in all tested configurations, consistently maintaining lower latency in non-quantized settings. At a 256K sequence length, MiniCPM-SALA achieves about 3.5× speedup over Qwen3-8B.

On longer sequences (512K and 1M long text), where same-size open-source models already hit VRAM blowups, MiniCPM-SALA still performs inference stably, making it possible to apply MiniCPM-SALA in more on-device domains such as cars, phones, and embodied robots.

In addition, with only 9B parameters, MiniCPM-SALA maintains excellent performance comparable to same-scale full-attention models in core capability dimensions such as knowledge Q&A, mathematical reasoning, and code generation, achieving comprehensive performance that is strong at both long and short contexts.

We invite you to push the limits of inference

To further explore the performance limits of the SALA hybrid attention architecture in real deployments, we have joined forces with SGLang, NVIDIA, and ModelBest to launch the 2026 Sparse Operator Acceleration Grand Prix (SOAR). The competition focuses on hardcore low-level topics such as sparse operator fusion and compiler optimization, challenging participants to achieve million-token inference and extreme KV Cache efficiency on consumer GPUs, and to jointly find the optimal performance solution for long-text processing. We sincerely invite technical geeks worldwide to push the limits based on the SGLang inference framework and NVIDIA hardware characteristics, break physical hardware bottlenecks, and jointly define the next performance baseline for efficient inference.

At 10:00 a.m. on February 12, we will break down the model, framework, and hardware scheduling bottlenecks across the full stack and provide an in-depth explanation of the competition tasks. Everyone is welcome to reserve a spot to watch; for more competition details, see the second article in our official account.

Technical report: 01 Introduction: As large models grow more capable, their application scenarios are undergoing a profound paradigm shift, from simple Q&A to increasingly complex tasks. In these advanced scenarios, models can no longer be limited to processing fragmented information. Instead, they must handle ultra-long contexts — for example, reading an entire technical manual in one pass, analyzing dependencies spanning tens of thousands of lines of code, or maintaining coherent task state and memory across days of human-AI collaboration. In this context, million-token processing will become a key capability for frontier large models. However, the Transformer architecture underlying today's mainstream large models faces a severe computational bottleneck when processing ultra-long sequences. This bottleneck manifests in two dimensions:

The compute wall (Compute Bottleneck): In standard attention, computational complexity grows quadratically with sequence length N. When context extends to a million tokens, the enormous overhead of the pre-filling stage causes time to first token (TTFT) to rise sharply.

The KV-Cache memory wall (Memory Bottleneck): During autoregressive generation, the model must store the Key and Value states of all historical tokens to avoid recomputation. For a typical 8B-parameter model, even with grouped-query attention (GQA), the KV-Cache required for a million tokens can reach tens or even hundreds of GB.

To address these challenges, existing solutions follow two main paradigms: sparse attention and linear attention. Both have unique advantages and inherent limitations. Sparse attention tries to break the compute wall by computing only the most salient parts of the attention matrix (for example, through sliding windows or global anchors). However, these methods are characterized by "sparse computation, dense storage." Although local computation reduces immediate processing overhead, the model still must retain the full KV-Cache to support retrieval of contextual information. Linear attention uses recurrent computation to successfully reduce complexity to O(N). However, this extreme efficiency is fundamentally achieved through lossy compression of contextual information, which easily degrades performance.

MiniCPM-SALA adopts a hybrid architecture of sparse and linear attention, designed specifically for efficient ultra-long sequence modeling. It combines InfLLM-V2's high-fidelity local modeling with Lightning Attention's global computational efficiency. Through this integrated approach, the model significantly reduces inference overhead and memory consumption while solving the accuracy bottleneck typical of pure linear architectures in long-range information processing.

Main contributions:

Innovative architecture design: We propose SALA, a sparse-linear hybrid attention mechanism combining 25% InfLLM-V2 with 75% Lightning Attention, striking a good balance between performance and efficiency. By leveraging sparse attention's fine-grained focus on local details and linear attention's O(N) efficiency at long context, the architecture maintains strong performance as sequence length grows.

Efficient training paradigm: MiniCPM-SALA demonstrates that converting a Transformer into a hybrid architecture is an efficient strategy for building hybrid models. By architecturally converting pre-trained weights, this approach avoids the inefficiency of cold-start training, with total training volume only 25% of training a comparable model from scratch.

Strong comprehensive performance: HyPE (hybrid position encoding) effectively reconciles short- and long-context performance. While matching modern full-attention models such as Qwen3-8B on general capabilities (knowledge, math, and coding), MiniCPM-SALA shows clear advantages on multiple long-context benchmarks.

Superior inference efficiency: MiniCPM-SALA shows significant resource and speed advantages in long-context scenarios. On an NVIDIA A6000D GPU at a sequence length of 256K tokens, MiniCPM-SALA runs up to 3.5× faster than Qwen3-8B. In addition, MiniCPM-SALA supports inference with context lengths up to 1 million tokens on NVIDIA A6000D and 5090 GPUs, while Qwen3-8B runs out of memory (OOM) at that length. These results demonstrate the strong potential of MiniCPM-SALA for on-device long-text applications.

02 Model architecture

(1) Overall architecture design

MiniCPM-SALA is built on an alternating hybrid attention architecture. As shown in Figure 1, unlike the homogeneous stacking of full-attention layers in a traditional Transformer, this architecture uses a heterogeneous layer design to break the performance bottleneck of a single mechanism in million-token long-sequence modeling. The model mixes 25% sparse attention with 75% linear attention. To significantly reduce compute cost, MiniCPM-SALA uses the HALO algorithm to convert a pre-trained full-attention Transformer into a hybrid architecture for continued training, rather than building from scratch.

Figure 1: MiniCPM-SALA model architecture

(2) Sparse attention module

MiniCPM-SALA uses InfLLM-V2 as the 25% sparse attention module. InfLLM-V2 is a switchable sparse attention framework whose core feature is a fine-grained block selection mechanism that lets each Query process only a small subset of key Keys and Values, greatly improving long-text processing efficiency. Thanks to InfLLM-V2's switchable nature, we enable sparse mode during long-text training and disable it during standard-length training (for example, 4,096 tokens). In small-scale experiments we verified that an output gating mechanism helps improve the model's general capabilities, so we also added output gating to the sparse attention layers.

(3) Linear attention module

For the linear attention component, although advanced linear token-mixing modules such as RWKV-7, GDN, and KDA have emerged, Lightning Attention's computational paradigm is closer to full attention and adapts better to the HALO conversion algorithm, so we ultimately chose it as the core operator for the linear layers. We also used the same architectural details as HypeNet, including QK-normalization, GQA-to-MHA transformation, and output gating; these settings effectively improve training stability and yield better model performance.

(4) Hybrid position encoding

We followed the HyPE (Hybrid Position Encoding) strategy proposed by HypeNet. It differentiates designs for the characteristics of different attention mechanisms in a hybrid architecture: in linear attention layers, we retain RoPE (Rotary Positional Embedding) to maximize consistency with the pre-conversion full-attention model in parameter distribution and feature space, thereby reducing performance loss from conversion.

In sparse attention layers, we adopt NoPE (No Position Encoding). The core advantage is that historical KV-Cache in sparse layers is no longer coupled with any positional information. This effectively avoids RoPE's inherent long-distance decay when processing ultra-long sequences, so when retrieving very distant historical information, the model is no longer constrained by numerical suppression from positional encoding, significantly enhancing long-distance recall.

03 Model training

The overall training process is shown in the table below:

Architecture conversion (HALO): This stage converts the Transformer model from full attention to a hybrid architecture, mainly following the HALO conversion method. Specifically, MiniCPM-SALA's training configuration differs from standard HALO in two ways. First, regarding layer selection, MiniCPM-SALA keeps the first and last layers unconverted to improve training stability. For the remaining layers, HALO's layer selection algorithm determines which layers remain full-attention. These retained full-attention layers are later trained as sparse attention in subsequent stages. The second difference from standard HALO is that we skip the final fine-tuning step in the HALO pipeline. Instead, we perform broader continued pre-training and post-training, as described in the following steps. This stage is very efficient, using only 1.3B tokens with a sequence length of 512 tokens. In addition, only the converted linear attention layers are trainable in this stage; all other parameters remain frozen.

Continued Stable training: Starting from the model at the end of the previous stage, we further train on the MiniCPM-4.0 pre-training dataset. The main goal is to help the converted linear attention layers better adapt to the rest of the model, including the full-attention layers, FFN layers, and embedding layers. The sequence length is set to 4K tokens, with a total training volume of 314.6B tokens. Because the sequence length is still relatively short, sparse attention layers are disabled in this stage to maintain computational efficiency. The learning rate is set to 7.5e-3.

Short-Decay training: In the third stage, the learning rate decays exponentially from 7.5e-3 to 3.75e-4. This stage uses a 4K-token sequence length and trains on 1T tokens in total, the largest training segment in MiniCPM-SALA's entire development pipeline. On the data side, building on the MiniCPM-4.0 Decay training data, we significantly increased the weight of high-quality L2 filtered data (for data tiering standards, see Data Science and Technology Towards AGI Part I: Tiered Data Management) and introduced a large PDF corpus and L3 synthetic data. This approach aims to strengthen general capabilities and logical reasoning through high-information-density data combinations, efficiently compressing and internalizing vast knowledge.

Long-Decay training: This stage gradually extends the context window from 4K to 32K, 160K, and finally 520K tokens, using 102.2B, 62.9B, and 50.6B tokens respectively. The learning rate decays from 3e-4 to 2e-4 at 32K, then to 1e-4 at 160K, and finally to 3.75e-5 at 520K. In this stage we upsample the proportion of long-context data so the model better aligns with long-sequence distributions. Given sparse attention's clear computational advantage on longer sequences, we enable sparse attention and keep full-parameter training in this stage, allowing the model to effectively learn the synergy between sparse and linear attention.

SFT: The SFT corpus in this stage consists of high-quality reasoning-intensive data, including code, math, knowledge, function calling, and general conversation. This choice aims to fully elicit reasoning and task execution under complex logic. In addition, we specifically synthesized long-context data to improve information retrieval and cross-document understanding. In the SFT stage, we train at context lengths of 64K and 140K in sequence, using 204.5B and 213.3B tokens respectively. Sparse attention remains enabled throughout.

04 Model capability evaluation

Short-context benchmark tests:

Long-context benchmark tests:

Ultra-long sequence tests:

These results show that MiniCPM-SALA effectively improves long-text capability without hurting short-benchmark performance. Another notable highlight is its excellent length generalization: without any additional techniques (such as YaRN), MiniCPM-SALA can extrapolate effectively to 2048K length.

05 Computational efficiency evaluation

We evaluated the inference speed of MiniCPM-SALA and Qwen3-8B on different hardware and sequence lengths. Experiments ran not only on cloud inference chips (such as NVIDIA A6000D) but also on consumer on-device GPUs (such as NVIDIA 5090). For each sequence length, we measured time to first token (TTFT) and end-to-end latency. The former indicates pre-fill speed, while the latter reflects combined pre-fill and decode performance. To keep the evaluation consistent with real deployment, we measured inference latency for both non-quantized models and models compressed with GPTQ INT4 quantization.

Figure 2 presents a comprehensive comparison of inference latency between Qwen3-8B and MiniCPM-SALA on an NVIDIA A6000D GPU (96GB VRAM). We evaluated efficiency at sequence lengths from 64K to 1024K tokens. As shown, MiniCPM-SALA exhibits a significant efficiency advantage in all tested configurations. In non-quantized settings, MiniCPM-SALA consistently maintains lower latency. Notably, at a 256K sequence length, MiniCPM-SALA reduces TTFT from 180.8 seconds (Qwen3) to just 51.6 seconds, achieving about 3.5× speedup.

The results also show MiniCPM-SALA's advantage in VRAM utilization. When Qwen3-8B hits out-of-memory (OOM) at 512K and 1024K sequence lengths, MiniCPM-SALA successfully handles these ultra-long contexts.

Figure 3 shows MiniCPM-SALA's advantage on hardware with limited VRAM. On an RTX 5090 (32GB VRAM), the Qwen3-8B baseline hits the "memory wall" much earlier than on A6000D, triggering OOM at only 128K tokens in non-quantized settings and 256K tokens with quantization. In contrast, MiniCPM-SALA successfully scales to a 1024K-token context without VRAM overflow. This shows that MiniCPM-SALA makes processing 1 million tokens possible on consumer on-device GPUs.

06 Conclusion

Overall, by combining sparse attention with linear attention, we built a hybrid architecture for efficient long-context modeling, achieving an excellent balance between model capability and long-context processing efficiency. Over the past period, research on sparse attention and linear attention has separately shown significant potential for solving large-model long-context problems; in 2026, the sparse-linear hybrid architecture will undoubtedly become one of the most practical technical directions in this field.