SOAR Weekly Champion Notes 07 | Vanilla Xiao Zhang: NVFP4 Quantization Improvements and Speculative Decoding in Practice
Editor's Note: Efficiency and Throughput
In this week's SOAR 2026 contest, team "Vanilla Xiao Zhang" took first place through deep refinement of quantization engineering and the inference pipeline. This retrospective focuses on improving NVFP4 weight quantization accuracy and adapting Medusa speculative decoding. By introducing the FourOverSix adaptive strategy, the team neatly solved the problem of insufficient representation density in specific numeric ranges under low-bit quantization. For MiniCPM-SALA's hybrid attention architecture, they also tackled the challenge of managing GLA recurrent states in the tree verify flow, achieving a steady increase in decode throughput.
From fine-tuning the quantization distribution to handling state branching in a complex architecture, this practice shows that high-performance inference comes not only from transplanting methods from individual papers, but from deeply deconstructing and rebuilding the details of model structure. This retrospective offers a highly practical engineering reference for large-model inference optimization.
Article at a Glance
This article reviews two things:
NVFP4 weight quantization accuracy improvement: reducing low-bit quantization error through FourOverSix adaptive block scale selection.
Medusa speculative decoding adaptation: making tree verify handle GLA recurrent states correctly on MiniCPM-SALA's hybrid attention architecture.
Quantization: FourOverSix adaptive block scale selection: 01 Background: Where standard NVFP4 quantization error comes from
Our baseline quantization route is:
GPTQ + NVFP4 (FP4 E2M1) + Marlin W4A16 decode; NVFP4's representable values are:
{0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}
Its maximum absolute value is 6. In standard NVFP4 quantization, each block's weights are normalized to the full FP4 range of [-6, 6], that is, with M=6 as the scaling upper bound.
This design has an obvious problem:
When the maximum weight value is mapped near 6, values in the 66%–100% range can only be represented by 6, and quantization error concentrates there.
02 FourOverSix: Adaptively choosing between M=6 and M=4
FourOverSix (Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, Song Han; a joint MIT + NVIDIA work, arXiv:2512.02010) has a very direct core idea:
For each quantization block, do not fix M=6; instead choose whichever of the two scaling upper bounds, M=6 or M=4, gives the smaller error.
With M=4, block weights are scaled to [-4, 4]. The [2, 4] interval can then use three values — 2, 3, and 4 — giving a more even representation distribution that suits blocks with concentrated weights.
The cost: M=4 gives up FP4's (4, 6] interval, reducing the ability to represent a few extreme values.
The selection rule is as follows:
## The block scale for M=4 is about 1.5x the M=6 scale; scale_m4 = fp8(scale_m6.float() * 1.5)
mse_m6 = ((W_block - dequant(W_block, scale_m6)) ** 2).mean()mse_m4 = ((W_block - dequant(W_block, scale_m4)) ** 2).mean()
final_scale = scale_m4 if mse_m4 < mse_m6 else scale_m6
The benefits of this approach:
The output format remains standard NVFP4: 4-bit weights + FP8 block scale
The inference kernel needs no changes
No extra throughput overhead
03 How we integrated it
We embedded adaptive scale selection into the GPTQ quantization flow:
Estimate block scale ↓Compare dequantization error for M=6 and M=4 ↓Choose the scale with smaller error ↓Enter GPTQ weight optimization iterations
The reason: fix each block's M choice first, then let the subsequent GPTQ optimization proceed within that framework, avoiding mutual interference between scale selection and GPTQ iterations.
04 Measured results
Measurements showed:
About 40–43% of blocks ultimately chose M=4
The M=4 share in MLP layers is clearly higher than in Attention layers' QKV projections
With FourOverSix enabled, stable quantization accuracy gains were observed across evaluation tasks
This is intuitive:
Speculative decoding: Adapting Medusa to a hybrid attention architecture: 01 The goal of speculative decoding
The basic idea of speculative decoding is:
Use a lightweight draft to predict several future tokens first, then have the main model verify them all at once, so the cost of one verify buys the chance to output multiple tokens.
If the predictions are accepted, the main model needs fewer decode steps, improving throughput.
02 Medusa: No separate draft model
Medusa (Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao; ICML 2024) has this core innovation:
Instead of training a separate draft model, add multiple lightweight prediction heads on the main model's last-layer hidden state.
Each Medusa head predicts the k-th future position.
The structure is:
p_t^(k) = softmax( W₂⁽ᵏ⁾ · ( SiLU(W₁⁽ᵏ⁾ · hₜ) + hₜ ) )
Where:
W₁ is initialized to all zeros. This makes each Medusa head's prediction identical to the main model early in training, stabilizing training.
03 Tree Attention Verify
The key to Medusa verify is validating multiple candidate paths at once.
The flow can be simplified as:
Multiple Medusa heads output candidate tokens ↓Take each head's top-s candidates ↓Build a candidate tree via Cartesian product ↓Main model verifies in parallel in one forward pass ↓Select the longest accepted prefix
For example, two heads each taking top-2 yield 4 candidate paths.
During verify, the attention mask is modified so that each candidate node can only see its own predecessor path, not other sibling branches. This verifies the entire candidate tree in one main-model forward pass.
04 The MiniCPM-SALA adaptation challenge: GLA state management
MiniCPM-SALA uses a hybrid attention architecture. Besides standard Multi-Head Attention, it contains many GLA (Gated Linear Attention) recurrent layers.
Each GLA step's output depends on the recurrent state hₜ:
hₜ = exp(−γ) · hₜ₋₁ + kₜ · vₜᵀ
This brings a new problem for tree verify:
Sibling branches in the tree must start from the same parent state; they cannot linearly inherit the previous candidate's state.
If a sibling branch wrongly inherits the previous candidate's state, it brings in history that does not belong to the current path, making verify semantically incorrect.
Our approach: add GLA state branching logic for verify paths, ensuring every branch starts from the correct parent state.
05 How training data distribution affects accept rate
Beyond architecture adaptation, the training data distribution for Medusa heads is also critical.
We observed that the closer the training data is to the online inference scenario, the more stable the accept rate. Based on this, we weighted the training data according to the validation set's distribution characteristics, bringing the training set closer to the actual inference scenario.
The result: compared with random sampling, accept rate improved measurably.
06 Experimental results
On MiniCPM-SALA:
Medusa K=1 verify overhead is about 0.39 ms
This overhead is clearly lower than the time of a normal decode step
At every concurrency level, end-to-end decode throughput shows a stable positive gain
Conclusion
What these two efforts share: you cannot simply copy methods from papers; you must understand the target model's structural details.
The key to FourOverSix is understanding the NVFP4 block scale format and its relationship with the GPTQ optimization flow.
The key to Medusa adaptation is understanding how GLA recurrent states differ from a standard KV cache.
Getting a method from a paper to run in a concrete system is often harder than reproducing the paper's results.
Thanks to the SOAR 2026 organizing committee for the platform, and to SGLang and OpenBMB for their open-source work. Thanks also to Claude and ChatGPT — as two undergraduates with no infrastructure background, we figured out many system-level issues only through repeated discussions with AI.
The competition continues, and we look forward to more exchanges with fellow contestants.
About the Authors
Wang Zhiyao: Team lead, fourth-year undergraduate in Artificial Intelligence at Huazhong University of Science and Technology. With no prior infrastructure background, he encountered systematic inference-system optimization for the first time through this competition.
Wu Xianjin: Teammate, fourth-year undergraduate in Artificial Intelligence at Huazhong University of Science and Technology. Used the competition as an entry point into GPU kernels.
Wu Xianjin would like to give special thanks to his girlfriend Zhang Yu for her company and understanding throughout the competition. Preparing for the contest meant late nights and occupied weekends, and he is grateful for her patience.
We are glad to have won Week 7 of SOAR 2026 together with our teammate.
We also sincerely thank the other teams from previous weeks for sharing their systems work. Those efforts gave us a clearer picture of the technical boundaries of this track and helped us judge faster which directions were worth pursuing.
🌟 Dear OpenBMB community:
The SOAR 2026 Sparse Operator Acceleration Grand Prix is in full swing, with new challenges every week and a live leaderboard waiting. If you want to show your technical strength in model inference acceleration, kernel-level optimization, and quantization, we sincerely invite you to enter and compete alongside Cao Yi and other outstanding participants, pushing the inference limits of MiniCPM-SALA together! Click the image below or visit the contest site at https://soar.openbmb.cn/ to learn more.