MiniCPM-SALA uses a sparse-linear hybrid attention architecture in which 25% of layers use InfLLM-V2 and 75% use Lightning Attention. It supports inference with up to a 1M-token context on a single NVIDIA A6000D (96GB) or a single RTX 5090 (32GB).
Highlights
Innovative hybrid architecture：Collaboratively integrating 25% sparse attention (InfLLM-V2) to achieve high-fidelity long context modeling, and 75% linear attention (Lightning Attention) to ensure global computing efficiency; compared with the dense baseline model, it achieves 3.5× inference acceleration and significantly reduces KV cache overhead, breaking the "computing power wall" and "video memory wall".
SOAR 2026 Sparse Operator Acceleration Race：The competition focuses on topics such as sparse operator fusion and compilation optimization. It focuses on million-token inference on consumer GPUs and on KV-cache efficiency for long-text processing.
Break efficiency bottlenecks：Inference runs at 3.5x the speed of the dense baseline with lower KV-cache overhead; test conditions and comparison data are given in the technical report.
Technical highlights
Why hybrid architecture is the better solution：When the traditional Transformer model processes long text, its core full attention mechanism will cause the amount of calculation to increase squarely with the length of the text. At the same time, the key-value pairs that need to be cached during the generation process will also occupy a large amount of video memory. Existing solutions are mainly divided into two major schools: linear attention and sparse attention. However, pure linear attention has the shortcoming of accuracy loss, and pure sparse attention has the limitation of storage bottleneck.
SALA hybrid attention mechanism：MiniCPM-SALA uses a sparse-linear hybrid attention architecture. 75% of layers use linear attention (Lightning Attention) for local information, and the remaining 25% use sparse attention (InfLLM-v2) for global information flow. Comparison data on compute efficiency and semantic accuracy under this ratio are given in the technical report.
Benchmarks
long context：MiniCPM-SALA consistently outperforms other open source large language models of the same size in most long context benchmarks: achieving the highest scores in the RULER and NoLiMa tests for all context lengths (up to 128K), demonstrating superior long context information processing capabilities.
Reasoning efficiency：Compared with the inference speed of Qwen3-8B, MiniCPM-SALA achieved up to 2.5× acceleration in the time to first token generation (TTFT), and it differs from full-attention architectures in memory usage: when Qwen3-8B encountered memory overflow (OOM) at the extended length, MiniCPM-SALA successfully expanded to millions of token contexts on a single consumer-grade RTX 5090.
Standard review：MiniCPM-SALA scores an average of 76.53 on the standard benchmarks; scores for the comparison models Qwen3-8B and Falcon-H1R-7B are given in the table below.
*OOM: Unable to run due to memory overflow (efficiency data based on single card actual measurement: RTX 5090/A6000D, non-quantified)