SOAR Weekly Champion Notes 05 | Zhisuan Team One: NVFP4 KV Cache and Mixed-Precision Practice
Editor's note · Low-bit and mixed precision
This week, in the SOAR semifinal, "Zhisuan Team One" once again demonstrated fine-grained engineering skill in inference optimization and took the championship. Through NVFP4 (FP4 E2M1) KV Cache exploration, they significantly reduced storage and bandwidth consumption in the decode stage and accelerated long-sequence inference. FP4 alone is not accurate enough, so the team performed layer-wise sensitivity analysis, keeping key layers in FP8 and using NVFP4 for the rest — achieving bandwidth gains close to pure FP4 while maintaining model accuracy.
This practice highlights the engineering wisdom of low-bit quantization: speed and accuracy require a careful trade-off, and mixed precision plus offline calibration are the key means to strike that balance. This retrospective offers a practical reference for developers focused on LLM inference optimization, and shows the engineering strategy behind Zhisuan Team One's win.
About the author: Xiaochen Li — a student focused on LLM inference system optimization, familiar with mainstream inference frameworks such as vLLM and SGLang, formerly an intern at a major tech company, with extensive hands-on experience in model precision quantization and KV Cache compression strategies. Teammate: Liu — currently responsible for GPU operator optimization at a company, familiar with Hopper and Blackwell architectures, with engineering experience optimizing GEMM and Attention on NVIDIA high-end GPUs. Below is the author's original share.
SOAR 2026 Semifinal
I'm glad to win the semifinal championship together with my teammate. This post looks back at our exploration of KV Cache compression during the semifinal stage.
01 Background: In earlier stages we had already completed basic optimizations such as W4A16 GPTQ + Marlin weight quantization and FP8 KV Cache. In the semifinal we wanted to push further on KV Cache compression and ease the VRAM bandwidth bottleneck in the decode stage, so we set our sights on a lower-bit NVFP4 KV Cache.
02 NVFP4 KV Cache: NVFP4 uses the FP4 E2M1 format (1 sign bit + 2 exponent bits + 1 mantissa bit), packing two FP4 values into one uint8 byte — halving storage and bandwidth overhead compared with FP8 KV Cache. Each quantization block has a scale factor, and dequantization simply multiplies by the scale to recover the value. This format suits the memory-bound nature of the decode stage: KV Cache reads are cut in half, which pays off especially in long-sequence scenarios. We adapted the full NVFP4 KV Cache pipeline in the SGLang framework, including FP4 memory pool management, attention backend adaptation, and the quantized write and dequantized read paths.
Once enabled, speed did improve, but accuracy dropped noticeably: compared with roughly 80% accuracy for FP8 KV Cache, pure NVFP4 KV Cache fell to about 75%, below the acceptable threshold.
03 Mixed-precision KV Cache: Pure FP4 is not accurate enough, pure FP8 is not fast enough, so we naturally turned to mixed precision — evaluating how sensitive each layer's K/V values are to final output accuracy, keeping FP8 KV Cache for sensitive layers and using NVFP4 for insensitive ones. Specifically, we ran layer-by-layer ablation, downgrading one layer's KV Cache from FP8 to FP4 in turn and observing the change in output accuracy to judge that layer's importance.
Experiments found that the model's first layer and last few layers are most sensitive to quantization precision, while middle layers are more robust. In the end we kept a few key layers in FP8 and used NVFP4 for all the rest, gaining bandwidth close to pure FP4 with only a small accuracy loss.
04 Summary: Exploring NVFP4 KV Cache taught us that low-bit quantization is not simply flipping a switch — it requires finding a balance between speed and accuracy. Mixed-precision strategies and offline calibration were the key means to reach that balance.
Acknowledgements: Thanks to the SOAR 2026 organizing committee for the contest platform and ongoing technical support, and to the SGLang team for their open-source work. The semifinal was fiercely competitive, and thanks to my teammate for the full cooperation. The experience gained in the contest has deepened our understanding of inference optimization, and we hope to keep doing meaningful work in the future.
🌟 Fellow members of the OpenBMB community: The SOAR 2026 Sparse Operator Acceleration Grand Prix is in full swing, with new challenges every week and a live leaderboard waiting for you. If you want to show your technical strength in model inference acceleration, operator-level optimization, and quantization, we sincerely invite you to enter the contest, compete alongside Cao Yi and many other outstanding participants, and together push the inference limits of MiniCPM-SALA! Click the image below or visit the contest website https://soar.openbmb.cn/ to learn more.