SOAR Weekly Champion Notes 06 | FlashSALA: Marlin GEMM Kernel Tuning and W4A16 Inference Optimization
Editor's note · Tuning and limits
This week in the SOAR 2026 weekly contest, FlashSALA took the weekly championship with systematic tuning of the Marlin kernel. This technical share focuses on Marlin GEMM kernel optimization in the SGLang framework: starting from fine-grained adaptation of the default general-purpose configuration, it designs multiple tile configurations for different M×N matrices and runs systematic experiments on pipeline depth, number of thread blocks, and other parameters. The author also optimized the atomic_add path in the decode stage. Through careful tuning of shared memory, registers, and thread configuration, the kernel in the W4A16 GPTQ inference scenario comes closer to the theoretical bandwidth utilization limit. This retrospective offers developers working on large-model inference optimization a practical tuning reference, and confirms that breakthroughs in high-performance inference depend on relentless attention to kernel details.
Below is the author's original share.
I'm glad to take this week's championship. In this post I'll share my work on Marlin GEMM kernel tuning.
01 Background: Marlin is the core GEMM kernel in SGLang for W4A16 GPTQ quantized inference. Through asynchronous pipeline prefetching and a clever data packing layout, it achieves near-theoretical bandwidth utilization at small batch sizes, making it a great fit for the memory-bound decode stage. But Marlin's default thread configuration is designed for general cases and is not finely adapted to specific model matrix shapes.
02 Tuning directions: Several key compile-time and runtime parameters affect Marlin kernel performance: pipeline depth, the number of thread blocks per SM, and tile shape. Together these determine shared memory usage, register pressure, and compute parallelism. I ran systematic experiments across all these dimensions.
03 Per-scenario tile selection: The original Marlin determine_exec_config only distinguishes large batch from small batch, but in real inference the M values span a wide range and the N dimensions of different linear layers vary greatly.
My approach splits tile selection into multiple tiers, each with its own priority-ordered table of candidate tile configurations. The core idea: in the decode stage the K dimension is longer, so it favors expanding along K to improve bandwidth utilization; in the prefill stage batches are larger, so it emphasizes parallelism along N. Each configuration also has to satisfy constraints such as shared memory not exceeding the limit and K and N being divisible. After extensive comparative experiments, I found the optimal tile configuration for each M×N combination.
04 Atomic add optimization on the decode path: In the original Marlin implementation, multiple thread blocks compute in parallel along N and then reduce to the final result. The original logic uses an N-dimension threshold to decide whether to use atomic_add reduction — only when ceil(M/64) × N ≤ 16384 does it take the atomic_add path, otherwise it falls back to barrier synchronization.
But when M is small, the serial waiting overhead of barrier synchronization becomes the bottleneck. I changed the decision logic so that small M unconditionally takes the atomic_add path, eliminating unnecessary synchronization overhead.
Acknowledgements: Thanks to the SOAR 2026 organizing committee and the SGLang team for their open-source work. Marlin kernel tuning is meticulous engineering spanning shared memory, registers, and thread configuration, and it requires repeated experimental validation. The contest is still going — best of luck to everyone!
About the author: Xiaochen Li — a student focused on LLM inference system optimization, familiar with mainstream inference frameworks such as vLLM and SGLang, formerly an intern at a major tech company, with extensive hands-on experience in model precision quantization and KV Cache compression strategies. A note: due to changes in team members, I created a new team, and the previous results listed under "Zhisuan Team One" have been removed from the leaderboard.
🌟 Fellow members of the OpenBMB community: The SOAR 2026 Sparse Operator Acceleration Grand Prix is in full swing, with new challenges every week and a live leaderboard waiting for you. If you want to show your technical strength in model inference acceleration, operator-level optimization, and quantization, we sincerely invite you to enter the contest, compete alongside Cao Yi and many other outstanding participants, and together push the inference limits of MiniCPM-SALA! Click the image below or visit the contest website https://soar.openbmb.cn/ to learn more.