SOAR 2026 Recap: From Compute Limits to the Open-Source Ecosystem, a Deep Conversation About the Future of Inference

2026-06-25

SOAR 2026, a large model inference optimization challenge lasting about three months, has officially drawn to a close.

The competition was jointly initiated by the OpenBMB open-source community together with the SGLang and NVIDIA communities. Centered on MiniCPM-SALA, the world's first "sparse + linear" hybrid architecture model, it focused on hardcore underlying topics such as sparsification kernel fusion and compilation optimization, challenging participants to achieve million-token inference and extreme KV Cache efficiency on consumer-grade GPUs. This article reviews the entire competition, documenting the contestants' technical evolution paths, the complete technical details of the champion solution, and the frontier talks at the closing Meetup.

About the SOAR 2026 competition

SOAR 2026 is a developer practice event for large model edge inference optimization, focusing on model performance optimization under real hardware, real inference frameworks, and real evaluation scenarios. The competition does not treat a single score as the only goal; rather, through complete engineering constraints, it aims to push participants to optimize comprehensively around inference efficiency, VRAM utilization, concurrency capability, and system stability.

The competition is based on the MiniCPM-SALA model. The model uses a 32-layer hybrid attention architecture, including 8 sparse attention layers and 24 linear recurrent attention layers, and supports ultra-long context of up to 512K tokens. The competition designates a single RTX PRO 6000D (Blackwell, sm_120, 84GB) as the evaluation hardware and uses SGLang as the inference framework. Evaluation covers three scenarios — single-request latency (S1), 8-concurrency latency (S8), and maximum-concurrency latency (Smax) — with a correctness gate set as a prerequisite.

This design means participants must not only focus on algorithm optimization at the model level but also dive deep into the engineering implementation of the inference system, including CUDA kernels, quantization toolchains, KV Cache management, inference engine scheduling, and hardware instruction-level adaptation.

Participation and technical evolution: 01 Overall data

Looking at overall participation data, the competition saw 326 teams register and 370 participants take part, with more than 4,300 total submissions, and 69 teams ultimately making the leaderboard. Many teams continued parameter tuning, solution validation, and engineering iteration throughout the competition period.

02 Technical evolution

The contestants' optimization paths showed a clear shift from basic parameter tuning to full-stack system optimization.

Early stage: contestants mainly experimented with quantization, KV Cache, inference parameters, and similar directions, such as W4A16 GPTQ quantization, FP8 KV Cache, and basic batch parameter adjustments.

Mid-to-late stage: leading teams began to go deep into the inference framework and low-level kernels, with more systematic optimization schemes such as kernel fusion, hybrid quantization routing, CUDA Graph capture, custom Triton / CUDA kernels, and speculative decoding.

Final stage: some top solutions involved code changes across more than a hundred files and over ten thousand lines of new code, covering model weights, attention computation, KV Cache, inference scheduling, and hardware execution paths.

Overall, the core optimization directions in this competition concentrated on five categories: low-bit quantization, KV Cache compression, kernel fusion and custom kernels, speculative decoding, and engine-level scheduling optimization. Together these directions show that large model inference optimization is no longer single-point parameter tuning, but coordinated optimization across model structure, inference framework, hardware capability, and system scheduling.

In terms of results, low-bit quantization increased compute density, KV Cache compression freed VRAM for high concurrency, kernel fusion and custom kernels reduced actual execution overhead, speculative decoding improved single-request latency, and engine-level optimizations such as CUDA Graph and torch.compile further reduced scheduling overhead. Together, these technical routes were the key to the leading teams pulling ahead in this competition.

The SOAR 2026 closing Meetup

The SOAR 2026 closing Meetup was the offline wrap-up of the competition. The event centered on "Large Model Edge Inference in Practice: From Compute Optimization to Open-Source Ecosystem Co-Building," covering competition review, award announcements, and champion solution retrospectives, while also inviting technical guests from NVIDIA, SGLang, and OpenBMB to share frontier progress from the three directions of hardware, inference frameworks, and models.

01 Opening: from the open-source community to the inference arena

The event opened with Jing Chenzhe, head of the OpenBMB open-source community, who introduced the positioning and value of the community and led the audience through a review of SOAR 2026's background, competition design, participation, and main technical results.

02 Technical talks: from hardware compute to inference frameworks, and on to on-device models

The technical talks at this Meetup centered on three directions: hardware, inference frameworks, and models.

Wu Zihua, NVIDIA accelerated computing expert, presented "SM120 Architecture Inference Optimization and Agent Practice," exploring how the hardware characteristics of the new-generation Blackwell architecture affect large model inference efficiency and the paths to application.

The SGLang team brought two topics: Alex shared, from a macro perspective, the evolution direction of open-source AI inference infrastructure over the next year, covering multi-hardware adaptation and ecosystem building; Ye Xuesong focused on SGLang-Omni, introducing this inference framework redesigned for multi-stage generative models (such as Qwen3-Omni). Centered on three core designs — scheduling decoupling, communication layering, and memory isolation — he systematically explained how to address the architectural challenges of multi-stage decoding of "thinking" and "expression." It is now open-sourced and supports frontier models including TTS, Omni, and Multimodal.

Xiao Chaojun, head of text models at OpenBMB, started from the macro perspective of Scaling Law and introduced the team's concepts of "capability density" and the Densing Law — the related research was selected as a cover article in Nature Machine Intelligence — and systematically described MiniCPM5's technical layout in three directions: compute-efficient architecture, high-knowledge-density data governance, and scalable reinforcement learning, showcasing the continuous evolution from the MiniCPM series to the latest "small but mighty" MiniCPM5-1B.

03 Moment of honor: the SOAR 2026 grand champion announced

After about three months of fierce competition and more than 4,300 submissions, the SOAR 2026 grand championship went to Team "Xiangcao Xiaozhang" (School of Artificial Intelligence and Automation, Huazhong University of Science and Technology; Wang Zhiyao and Wu Xianjin).

Score data

Compared with the Baseline, the champion solution achieved an overall average speedup of 6.33x, performing strongly across all three core evaluation scenarios:

S1 (single request): speedup 9.72x (1029.49s → 105.93s), nearly 10x faster;

S8 (8 concurrency): speedup 5.42x (1298.65s → 239.77s);

Smax (maximum concurrency): speedup 3.86x (2544.34s → 659.34s).

The champion solution delivered the most significant speedup in the single-request scenario; the speedup converged somewhat as concurrency scaled up, but still maintained nearly 4x improvement at maximum concurrency — showing that the solution optimized not only single-point latency but also system throughput and stability under high concurrency.

A deep dive into the champion technical solution

The champion team's final solution covered three core directions — quantization, attention backend, and speculative decoding — with each direction reaching down to the hardware instruction level.

1. Quantization and GEMM dispatch

The champion solution uniformly quantized all linear layer weights to NVFP4 (FP4 E2M1, 0.5 bytes per element). The quantization process uses GPTQ's Hessian-optimal rounding and introduces FourOverSix block-adaptive scale selection: each quantization block can flexibly switch between scale=4 and scale=6 granularities. In practice, about 40–43% of blocks chose the finer scale=4, noticeably improving quantization error in MLP layers.

On the GEMM path, the team implemented a Hybrid Marlin/CUTLASS dynamic dispatch strategy. When the number of tokens in a single GEMM is M ≤ 48, it uses the Marlin W4A16 implementation; when M > 48, it switches to the CUTLASS W4A4 path, natively invoking NVFP4 MMA instructions to maximize compute density. The latency curves of the two paths cross exactly around M ≈ 48, and the runtime selects the lower-latency implementation. In addition, KV Cache keeps BF16 precision throughout to preserve overall inference stability.

2. Hybrid attention backend optimization: Standard Attention: FlashInfer Plan Cache

A single plan() call takes about 23ms; the baseline 128K prefill requires 128 calls (16 chunks × 8 layers), totaling 2.9s, or 20%.

With a fixed top-K, the number of blocks per layer matches the KV segment length; a two-level cache is designed (cross-layer within a forward + cross-forward reuse across chunks).

On a hit, only the index pointers are replaced; plan() calls drop to 16, and a single 128K prefill is shortened by 28% (14.6s → 10.3s).

GLA linear attention

Direct write to the state pool: fuse the three steps of gather → kernel → scatter (two full state copies) into indexed reads and writes inside a Triton kernel, eliminating outer data movement.

Kernel fusion: QKNorm+RoPE (5→1 kernel), output gating RMSNorm+sigmoid×mul (3→1 kernel), with computation completed in registers, eliminating HBM round trips for intermediate tensors.

3. EAGLE-3 speculative decoding and hybrid architecture adaptation

The core idea of speculative decoding is that a smaller draft model proposes multiple candidate tokens at once, and the target model then verifies these candidates in parallel in a single forward pass, accepting only the longest prefix consistent with the original autoregressive decoding result. Compared with token-by-token decoding, this method can produce multiple tokens in one target forward, thereby reducing decoding latency in the single-request scenario.

EAGLE-3 implementation

This solution uses EAGLE-3 as the speculative decoding framework. Specifically, EAGLE-3 extracts hidden states from three intermediate layers of the target model, concatenates them, and feeds them into a draft head of about 437M parameters to predict subsequent tokens:

Draft training

Loss: the draft acceptance rate satisfies; maximizing acceptance is equivalent to minimizing the TV distance between the draft and target output distributions. But pure TV has flat gradients early in training, while KL over-penalizes the low-probability tail. adaptively weights the two based on the current acceptance rate — KL warm-starts the early phase, and as it rises, the objective transitions to TV, which is dual to the acceptance rate.

AOI position augmentation: training samples are only 4K tokens, yet inference must handle ultra-long context. Anchor-Offset Interpolation gives each sample a random global position offset, so the draft sees RoPE at various position ranges during training and can extrapolate to ultra-long sequences (the first few positions of the sequence are kept as attention sinks and do not participate in the offset).

NVFP4 QAT: the draft must run 5–7 steps per target forward cycle, with latency added directly to the critical path, so quantization-aware training is applied — the forward pass performs GEMM directly in FP4 (rather than simulating fake quantization), and the backward pass passes gradients straight through to the BF16 master weights via STE, aligning training with the inference FP4 path and keeping weights consistent with inference.

Runtime scheduling

Candidate tree: tree verify verifies in parallel.

Adaptive speculation depth: D7 (15 candidates) for single requests, D5 (11 candidates) at medium concurrency, and a fallback to regular decoding at high concurrency; threshold hysteresis avoids repeated switching.

N-gram routing: deterministic fragments are matched as strings within the request; on a hit, a single chain is constructed to skip the draft, and on a miss it goes through EAGLE, sharing the same verify.

Long-context prefill truncation: leveraging chunk independence, only the last 32K performs draft prefill, cutting draft prefill time from 17s to ~1s with no significant drop in acceptance rate.

4. Tree verify adaptation for MiniCPM-SALA

MiniCPM-SALA has 32 layers in total, alternating between 8 sparse attention layers and 24 SimpleGLA layers. EAGLE speculative decoding uses tree verify to check the entire candidate tree in parallel in a single forward pass, but the state semantics of both layer types are disrupted by the tree structure, so standard EAGLE verification cannot be applied directly.

At the same time, speculative decoding must remain lossless — that is, with acceleration enabled, model outputs must still be exactly identical to native token-by-token decoding. The champion solution therefore adapted both the sparse attention layers and the SimpleGLA layers specifically.

Sparse attention layers

During decoding, each query selects only the top-k blocks from the historical KV to participate in attention. If verification degrades to full attention, the block selection process differs and the output deviates from the true decoding path.

Solution: each candidate node independently replicates the sparse attention block selection process used during decoding, and then a Triton kernel compresses the tree mask (which allows seeing only ancestor nodes and excludes sibling nodes) into a compact page table, so that each candidate accesses exactly the blocks it should see in real decoding.

SimpleGLA layers

SimpleGLA layers do not keep a KV cache; they maintain only a recurrent state that is irreversibly updated with each token, with a fixed scalar decay coefficient γ per head (data-independent, 0< γ <1):

During tree verification, after the entire candidate tree is rolled out once, the recurrent state is polluted by all paths (including branches that are later rejected) and must be rolled back to the correct state corresponding to the accepted path.

Option 1: state snapshots

Keep an independent full state snapshot for each candidate node. It is simple to implement and slightly faster, but at deployment scale the VRAM overhead is as high as about 8.8 GB, which is unacceptable.

Option 2: in-register replay (adopted)

Keep no intermediate state at all and store only the k, v inputs of each candidate node (about 0.14 GB of total VRAM). The verification forward pass allocates one thread per (sequence, token, head, state chunk), starting from the live initial state and replaying the recurrence step by step along the parent chain in registers to compute that candidate's state and output. After verify determines the accepted length, replay once more along the accepted path to generate the final state, and write back only that one state. State never touches VRAM throughout, and persistent VRAM usage is more than an order of magnitude lower than the full-state snapshot approach.

Conclusion

Three months of competition, more than 4,300 submissions — what SOAR 2026 leaves behind is more than a leaderboard.

In this competition, contestants truly faced the complex engineering problems in large model inference systems: how to compress VRAM, reduce latency, and increase throughput under limited hardware resources; how to make model structure, inference framework, and hardware capability work together; and how to push a performance optimization from "it runs" to "it runs faster, more stably, and more reusable." These are also exactly the questions that must be answered as on-device models and efficient inference move toward real applications.

SOAR 2026 has ended, but the exploration around efficient inference will continue. Going forward, the OpenBMB community will keep fostering collaboration among more models, tools, frameworks, and developers, so that more optimization solutions from the community become real, usable open-source results.