SOAR Weekly Champion Notes 08 | Vanilla Xiao Zhang: Evolving Speculative Decoding on MiniCPM-SALA
Editor's Note: Speculation and Throughput
In this week's SOAR 2026 contest, team "Vanilla Xiao Zhang" took first place again. This retrospective focuses on the ongoing evolution of speculative decoding on MiniCPM-SALA. From Medusa to EAGLE, the team systematically analyzed how different draft architectures differ in accept rate, verify overhead, and long-context scenarios, and further discussed the diminishing returns of speculative decoding under high concurrency.
From the capacity bottleneck of Medusa's multi-prediction-head structure, to the accept rate gains from EAGLE's autoregressive draft, to engineering observations in long-context and large-batch scenarios, this retrospective asks not only "how to accelerate" but "when acceleration actually works." For developers working on large-model inference systems and real-world speculative decoding, it is a reference with both theoretical analysis and engineering practice.
I'm glad to have won Week 8 again with my teammate.
Last week's write-up covered our quantization scheme and the initial Medusa adaptation. This article continues that thread, discussing the migration from Medusa to EAGLE and several issues observed in long-context and high-concurrency scenarios.
01 A Brief Recap of Speculative Decoding
The decode phase of large models is constrained by memory bandwidth: generating each token requires moving the full weight matrix from GPU memory once, leaving compute units heavily underutilized. Speculative decoding targets this bottleneck: a lightweight draft model generates K candidate tokens in advance, and the target model verifies them in parallel in a single forward pass. With the output distribution guaranteed to remain exactly unchanged, one forward pass can produce multiple tokens.
The actual gain depends on the cost of generating each draft candidate, the fraction of candidates accepted by the target (accept rate), and the extra time introduced by verifying K candidates. Together these three determine whether speculative decoding yields a positive return under a given deployment setting.
02 Medusa: Adding Prediction Heads to the Main Model
Medusa (Cai et al., ICML 2024) does not introduce a separate draft model. Instead, it attaches K lightweight MLP heads to the target's last-layer hidden state, where the k-th head predicts the token distribution at the k-th future position:
p_t^(k) = softmax( W₂⁽ᵏ⁾ · ( SiLU(W₁⁽ᵏ⁾ · h_t) + h_t ) )W₁ is initialized to a zero matrix, so early in training each head's output is equivalent to the main model, which helps convergence stability. In the verify stage, the top-s candidates from each head form a candidate tree via Cartesian product, and are verified in a single forward pass with a tree attention mask. The engineering advantages of this design are a small parameter increase, no autoregressive draft inference, and a simple training pipeline.
But Medusa's structural ceiling stems from the same design: each head predicts independently, so the second head cannot see the first head's prediction, and there is no recurrent context shared across heads. Each head is essentially a shallow mapping of the same hidden state, and accuracy decays more sharply the further ahead it predicts. Meanwhile, as K grows, the candidate tree expands exponentially, and verify overhead grows faster than accept rate improves.
We implemented Medusa with K=1 on MiniCPM-SALA and achieved a positive end-to-end gain. But as the number of heads increased, the accept rate curve quickly flattened — this reflects an inherent capacity limit of the architecture, not an implementation issue.
03 EAGLE: An Autoregressive Draft Model
EAGLE (Li et al., arXiv:2401.15077) makes a fundamental change to the draft architecture: the draft itself is an autoregressive model whose input is the target's hidden state rather than token embeddings. The draft receives compressed semantic features from the target's intermediate layers and approximates its output distribution with far fewer parameters than the target. Unlike Medusa, the draft performs K autoregressive steps itself, each taking the previous step's output as input, so it has a coherent recurrent context — precisely what Medusa's heads lack.
After migrating from Medusa to EAGLE, the accept rate ceiling improved substantially.
04 Accept Rate Decay in Long Context
In our measurements we observed a consistent phenomenon: under long-context conditions, speculative decoding's accept rate declines systematically. This is not specific to any one draft architecture but the result of several mechanisms stacking.
The first is positional encoding extrapolation. Draft models are usually trained mostly on shorter contexts (due to training cost). When the inference sequence length significantly exceeds the training distribution, positional embeddings enter the extrapolation regime, the draft's internal representations drift from those seen in training, and prediction accuracy drops accordingly.
The second is draft KV error accumulation. When the draft runs K autoregressive steps, each step depends on the KV cache produced by previous steps. With short contexts, the deviation between these KVs and the target's remains within verify's tolerance; with long contexts, the KV sequence grows, small per-step deviations amplify along the sequence, and by the end of K steps the draft's internal state may have drifted significantly from the target's.
05 Returns Revert Under High Concurrency
The gain from speculative decoding shrinks as batch size grows and turns negative once the batch is large enough. This has been discussed in the academic literature (Sequoia, SpecInfer, etc.), but it is still easy to overlook in engineering deployments.
At small batch sizes, GPU compute is severely underutilized: processing 1 token versus K+1 candidates costs the target almost the same, because the bottleneck is memory bandwidth rather than compute, so verify's marginal cost is near zero. At medium batch sizes, GPU utilization rises, verify incurs real compute cost, and the target's single forward must process batch × (K+1) tokens in parallel, so the extra time gradually approaches the token-output gain. At large batch sizes, the GPU is nearly compute-bound, forward time scales roughly linearly with total tokens, and speculative decoding trades batch × (K+1) compute for at most K extra tokens; when the accept rate cannot cover that cost, turning speculative decoding off is the better end-to-end choice.
✒️About the Authors
Wang Zhiyao: Team lead, fourth-year undergraduate in Artificial Intelligence at Huazhong University of Science and Technology.
Wu Xianjin: Teammate, fourth-year undergraduate in Artificial Intelligence at Huazhong University of Science and Technology.
🌟 Dear OpenBMB community:
The SOAR 2026 Sparse Operator Acceleration Grand Prix is in full swing, with new challenges every week and a live leaderboard waiting. If you want to show your technical strength in model inference acceleration, kernel-level optimization, and quantization, we sincerely invite you to enter and compete alongside Cao Yi and other outstanding participants, pushing the inference limits of MiniCPM-SALA together! Click the image below or visit the contest site at https://soar.openbmb.cn/ to learn more.