SOAR Weekly Champion Notes 04 | Zhisuan Team One: W4A16 Quantization and Inference Kernel Fusion in Practice
Editor's Note: Engineering and Efficiency
This week, the SOAR contest saw a major breakthrough. With a rigorous engineering approach, Zhisuan Team One ended a three-week championship streak and unlocked the contest's "Special Bounty Award." The result marks a shift in contestants' optimization thinking — from early strategy experiments to a combined application of deep kernel tuning and high-performance quantization.
In this edition, Zhisuan Team One walks through how they built a high-confidence local evaluation set to iterate efficiently within a limited number of submissions. On the technical side, they compared quantization paths in depth, ultimately easing the bandwidth bottleneck with a W4A16 (GPTQ + Marlin) scheme and squeezing further inference performance through kernel fusion.
With the semifinals approaching on April 8, the competition has become a test of overall engineering capability. This practical recap of quantization choices, kernel tuning, and local infrastructure offers a grounded engineering reference for any developer working on LLM inference optimization.
About the author: Xiaochen Li is a student focused on LLM inference system optimization. He is familiar with mainstream inference frameworks such as vLLM and SGLang, has interned at a major tech company, and has extensive hands-on experience in model precision quantization and KV Cache compression strategies. Teammate: Liu, who currently leads GPU kernel optimization at a company, is familiar with Hopper and Blackwell architectures and has engineering experience optimizing GEMM and Attention on high-end NVIDIA GPUs. Below is the author's original account.
SORA 2026: Week 4
I'm glad to have won the weekly championship with my teammate in Week 4. From the start of the contest until now, we've hit pitfalls and tuned our way through. This article reviews the main work we've done over the past few weeks.
01 Infrastructure: After receiving the problem statement, we didn't rush into optimization; we first laid a solid engineering foundation. We got the full SGLang + MiniCPM-SALA inference pipeline running locally. Given only three submissions per day, we needed a reliable local evaluation method to filter out unreliable ideas and avoid wasting online submissions. So we built a local evaluation set following the length distribution of the official speed benchmark set; our online scores and local results were largely proportional. We also analyzed the official accuracy evaluation set, looking at the distribution of task types and input length characteristics, which gave us a basis for preparing calibration datasets later.
02 Kernel fusion: My teammate specializes in kernel optimization on NVIDIA GPUs, so before the quantized model was ready, we reviewed the model's forward computation, identified kernels that could be fused or tuned, and rewrote some of them. For example, the RMSNorm normalization + RoPE rotary position embedding in the original code is a typical fusion candidate; the SGLang library provides corresponding fused kernels we could adapt and use, and fusing these brought a measurable gain.
03 Quantization: Quantization was where we invested the most. We first tried W8A8 dynamic quantization — quantizing both weights and activations online to INT8, hoping to use INT8 Tensor Cores for acceleration. The result was disappointing: in our scenario, speed barely changed.
Next we tried W8A16, quantizing only weights to 8-bit while keeping activations in BF16. Accuracy was almost unaffected, but the performance gain was limited and not enough to separate us from the field.
We ultimately chose W4A16 with GPTQ + Marlin. Compared with simple round-to-nearest quantization, GPTQ preserves higher accuracy at the same bit width. The inference kernel uses Marlin, which at small batch sizes approaches the theoretical 4x bandwidth saving — a great fit for the memory-bound decode phase. With W4A16, model size shrank to about one quarter of the original, the bandwidth bottleneck eased noticeably, and end-to-end latency dropped significantly across concurrency levels.
Calibration data has a substantial impact on quantization accuracy, and we iterated on it several times, adjusting the data mix according to the task distribution of the evaluation set.
04 Summary: Overall, solid infrastructure helped us a lot. Only with good infrastructure can you iterate and validate models quickly, and run repeated engineering verification and tuning. There is still a lot to do next, and I hope we can build more interesting things.
Acknowledgements: Thanks to the SOAR 2026 organizing committee for the competition platform and continuous technical support, and to the SGLang team for their open-source work. Seeing the leaderboard change every week reminds me how fast the other contestants are improving — pressure and motivation at the same time. The contest continues, and I wish everyone better results!
🌟 Fellow members of the OpenBMB community:
On March 28 (this Saturday), we are co-hosting an offline Infra AI Meetup with SGLang and GPUStack. Come meet the core members of MiniCPM, SGLang, and GPUStack on site and talk through everything about AI Infra. Click the image below to learn more!