SOAR Weekly Champion Notes 03 | Cao Yi: Quantization Accuracy Stability and Dataset Analysis
Editor's Note: Depth and Steady Progress
This week, contestant Cao Yi defended his title again, completing a three-peat in the SOAR contest. As quantization optimization approached the performance ceiling, he did not keep blindly chasing speed. Instead, he turned to the more fundamental work of accuracy stability and engineering infrastructure, focusing on accuracy jitter, the evaluation workflow, and calibration data.
The champion's progress toward the top bounty award has now reached 97%, with the grand prize close at hand — every step forward matters more than ever. From earlier mixed-precision quantization to this week's focus on accuracy jitter control and a fast local evaluation tool, he turned unstable submissions that relied on luck into controllable, predictable optimization with a reproducible, lightweight evaluation scheme. He also analyzed the evaluation dataset in depth and redesigned the quantization calibration strategy, laying a solid foundation for pushing the score higher.
As the contest progresses and the semifinals approach, competition has shifted from raw performance squeezing to a combined test of accuracy, stability, and engineering capability. Whether you are a contestant climbing the leaderboard or a developer interested in large model inference quantization, this week's hands-on experience with accuracy evaluation, dataset calibration, and quantization stability offers practical ideas.
About the Author: Cao Yi (SlightwindSec)
Cao Yi currently works at a domestic computing platform as an engineer for LLM inference framework development and model performance optimization. With more than two years of frontline engineering experience, he focuses on key areas such as inference acceleration and low-bit quantization. Below is the author's original account.
SORA 2026: Week 3
Very lucky to win three weeks in a row. Quantization work feels like it has hit a bottleneck here. This week I barely pushed forward; instead, I spent some time adjusting the calibration dataset and doing some foundational work.
01 Accuracy jitter
Last week I shared that I encountered unstable accuracy with quantized inference: the same code and the same configuration, but different submission batches would jitter in correctness — with bad luck, you absorb the accuracy penalty coefficient.
With only three submissions per day, unstable results waste submission opportunities, so the jitter had to be fixed (or accuracy improved). Since I started with the simplest calibration dataset, there was room to optimize it.
Evaluating quantization quality is a tricky business:
Running generative tasks reflects real inference quality well, but it is time-consuming; if there is jitter, you need multiple runs and averages, doubling the time cost.
MMLU runs first-token accuracy quickly, but it differs from generative task scenarios, and flip phenomena mean the accuracy result cannot truly reflect the gap between the quantized model and the floating-point model[^1]. First-token accuracy may well be "false prosperity."
So we can borrow the idea from "Accuracy is Not All You Need" and compare the logits distribution over the output tokens. SGLang can expose logprobs, so without modifying SGLang's source we can compare logprob distribution gaps — well suited to quick local accuracy validation before submission, and the results are stable.
02 Fast logprob-based accuracy evaluation
The core idea of the local accuracy evaluation tool is to compare the logprobs distribution of the quantized model against a BF16 baseline, rather than running the full benchmark suite.
The flow has two stages:
The first stage collects the baseline, which only needs one run: use the BF16 model to greedily decode a set of built-in prompts, extract the top-256 logprobs of the last 128 tokens, and save them as sparse baseline files for batch sizes 1/8/32.
The second stage tests the accuracy of the quantized weights: start the service, send the same prompts, compare against the baseline, and output three metrics:
The lower the KL divergence and the higher the cosine similarity, the closer the quantized model's output distribution is to BF16, and the smaller the accuracy loss.
Several design details are worth noting:
Top-256 sparse representation: SGLang does not support returning full-vocabulary logprobs, so we approximate with top-256, which covers >99.9% of the probability mass, with the remaining probability distributed evenly for smoothing.
Zero GPU usage: baseline data is stored purely with numpy and loaded into CPU memory. Evaluation only sends requests to the service, making it very lightweight and runnable at any time.
There is one known limitation: the tool can only sense accuracy error in the Prefill stage and does not cover error propagation during Decode.
With this tool, we can quickly verify the accuracy state of the current configuration before each submission and screen out clearly abnormal approaches in advance.
03 Evaluation dataset analysis
I did some simple analysis of the official accuracy evaluation dataset: 150 entries in total, 30 each across 5 task types:
We can use this data ratio, with some adjustments, to build a suitable quantization calibration dataset. Note that the test dataset has a high share of data with low semantic relevance. When building the calibration dataset, we can appropriately reduce that share and replace it with more data with continuous semantics — too much random-word and UUID data may negatively affect calibration.
04 Using NVFP4 Quantization
I am currently quantizing weights in NVFP4, a 4-bit floating-point quantization format introduced with NVIDIA's Blackwell architecture. The contest's evaluation machine is a Blackwell GPU with native hardware acceleration, which is the main reason for choosing it.
In terms of format definition, NVFP4 uses an E2M1 structure (1 sign bit, 2 exponent bits, 1 mantissa bit) and can represent values roughly in the range of -6 to 6. Its two-level scaling mechanism captures the local distribution of weights more accurately and preserves more accuracy at the same compression rate:
Every 16 weights form a group sharing an FP8 (E4M3) scaling factor for local correction.
The entire tensor then applies a global FP32 scaling factor.
05 This week's progress
This week's progress was mainly: building a fast local accuracy evaluation tool, analyzing the evaluation dataset distribution, and optimizing the calibration dataset.
There are more directions worth exploring, and I look forward to a bigger breakthrough next week.
Acknowledgements: Thanks to the SOAR 2026 organizing committee not only for an excellent competition platform but also for patient answers and solid support throughout the contest; thanks also to the SGLang team for their open-source contributions. Every change on the leaderboard shows me everyone's formidable technical strength. Competition is not just rivalry — it is mutual learning. The contest still has a long way to go, and I wish all contestants can keep pushing their limits and optimize toward more perfect results! I will keep optimizing too, striving for new breakthroughs.
[^1]: Accuracy is Not All You Need, Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, Ramachandran Ramjee
🌟 Fellow members of the OpenBMB community:
The SOAR 2026 Sparse Operator Acceleration Grand Prix is in full swing, with new challenges every week and a live leaderboard waiting for you. If you want to show your technical strength in model inference acceleration, operator-level optimization, and quantization, we sincerely invite you to enter the contest, compete alongside Cao Yi and many other outstanding participants, and together push the inference limits of MiniCPM-SALA! Click the image below or visit the contest website https://soar.openbmb.cn/ to learn more.