SOAR Weekly Champion Notes 02 | Cao Yi: How I Played Assistant to the AI Teacher and Pushed the Progress Bar to 96%
Editor's Note: Progress and Witnessing
This week, we witnessed Cao Yi defend his title. If the first week broke through quickly with "weight quantization," this week he went a step further into the details of mixed-precision quantization.
The champion's progress toward the top bounty award has now reached 96%. The last 4% is often the hardest challenge, testing how contestants can squeeze performance while holding the "accuracy red line." Will we witness the grand prize unlock next week? Worth looking forward to!
Beyond digging into performance optimization, this edition has another highlight: Cao Yi shares in detail how he efficiently assisted the AI teacher with coding. When handling complex kernel optimization and engineering adaptation, he developed a structured instruction-feedback workflow that let AI truly become a productivity "co-pilot," freeing him to focus on core architecture thinking.
Whether you are a contestant pushing the leaderboard or a geek interested in AI-assisted development, this edition's "tooling" experience should offer a different kind of inspiration.
About the author: Cao Yi (SlightwindSec) currently works at a domestic computing platform as an LLM inference framework developer and model performance optimization engineer. With more than two years of frontline engineering experience, he focuses on inference acceleration, low-bit quantization, and other key technical areas. Below is the author's original share.
SORA 2026: Week 2
Very honored to defend the weekly championship. The opponents are all strong; winning was not easy.
With limited energy for the contest outside work, this time I mainly "assisted" the AI: it conceived the direction and implemented it, taking on most of the workload, and I learned a lot in the process. This article shares this AI-led inference performance optimization workflow and what I took away from it.
01 Contest rules recap
The core goal of the contest is to minimize inference latency while ensuring model accuracy.
The final score is calculated as: (S₁×40% + S₈×30% + S∞×30%) × correctness coefficient C. Here S₁, S₈, and S_∞ are the latency scores under single concurrency, 8-way concurrency, and unlimited concurrency, respectively. The correctness coefficient C is a strict step penalty: below 97% accuracy scores zero; only at 99% or above do you get the full 1.0 coefficient; in between it is discounted proportionally.
02 Model structure code optimization
After analysis with Claude Code, I found room to squeeze performance in the model structure code. For example, the LayerNorm operation can be fused with the residual connection; here we can directly reuse SGLang's existing fused kernels. In addition, frequent type conversions in the code also add latency overhead. The original code converts q and k to FP32 for the rotary_emb computation — written this way to ensure higher accuracy, but optimizing it away carries a risk of accuracy loss:
orig_dtype = q.dtype
q, k = q.float(), k.float()
q, k = self.rotary_emb(positions, q, k)
q, k = q.to(orig_dtype), k.to(orig_dtype)
03 Mixed-Precision Quantization
Last week I used KV Cache FP8 quantization, which had minimal impact on model accuracy and brought a nearly "free" throughput gain. However, 4-bit weight quantization further raised throughput but caused a fairly large accuracy drop.
To solve the accuracy problem, I tried other quantization approaches and used mixed precision in actual inference, reducing latency and raising throughput as much as possible while keeping accuracy loss acceptable.
04 AI workflow: Gemini and Claude working together
For this contest I mainly relied on Gemini 3.1 Pro (chat and Deep Research) and Opus 4.6 (Claude Code). The overall collaboration flow was as follows:
1. Choose the optimization direction: First, organize the detailed background and send it to Gemini 3.1 Pro, asking it to list feasible optimization suggestions. After choosing the direction I want to try, have Gemini break it down further and provide relevant references and open-source repositories.
2. Refine the prompt: Once the direction is clear and initial information gathered, start drafting the prompt for Claude Code. To make the instructions clearer, I usually have Gemini polish the prompt, straighten out the logic, and make the description clearer.
3. Serve tea and water to Claude Code: Prepare the reference text materials and code repositories under the local path, and add corresponding explanations in the prompt or CLAUDE.md, giving Claude Code ample conditions to execute.
4. Code implementation and test loop: Finally hand the task to Claude Code. I require it to write lightweight latency and accuracy test scripts while modifying code. Since performance optimization must be premised on meeting accuracy, I have it rerun tests after every change, forming a complete optimization loop.
Here is a simple example:
**Task goal**
Based on the `sglang` fork in the current environment (which supports the new `MiniCPM-SALA` model), perform general performance analysis and code optimization.
**Performance evaluation criteria**
Evaluate performance by running `python scripts/bench_offline.py` and checking the table at the end of the output. Focus on the following $3 \times 2 = 6$ combined scenarios:
* **Data type**: long input, short output (reflects Prefill performance); short input, long output (reflects Decode performance).
* **Batch Size (BS)**: 1, 8, 64.
**Constraints**
1. **Stability and generalization**: The current test cases are simplified static batches, but your optimization must be general. Absolutely do not break the library's existing normal functionality, and do not degrade performance in real serving scenarios (continuously sending complex requests).
2. **Mandatory configuration**: `"disable_radix_cache": True` must be kept (radix cache must never be enabled).
3. **Allowed operations**: You may modify source code, change configuration to enable features, write performance collection scripts, or use tools such as `nsys` to assist performance analysis.
**Workflow requirements**
1. **Ask before acting**: Don't overthink. Before each code change or time-consuming test, briefly tell me your plan and **act only after I agree**.
2. **Documentation**: After each action and measured result, record the operation in the summary document. Keep it minimal: 1-2 lines on what was changed, and one sentence summarizing the test result (record it whether or not performance improved).
Please begin your exploration and optimization according to the above requirements. What do you plan to do first?
Acknowledgements: Thanks to the SOAR 2026 organizing committee not only for an excellent competition platform but also for patient answers and solid support throughout the contest; thanks also to the SGLang team for their open-source contributions. Every change on the leaderboard shows me everyone's formidable technical strength. Competition is not just rivalry — it is mutual learning. The contest still has a long way to go, and I wish all contestants can keep pushing their limits and optimize toward more perfect results! I will keep optimizing too, striving for new breakthroughs.
🌟 Fellow members of the OpenBMB community:
The SOAR 2026 Sparse Operator Acceleration Grand Prix is in full swing, with new challenges every week and a live leaderboard waiting for you. If you want to show your technical strength in model inference acceleration, operator-level optimization, and quantization, we sincerely invite you to enter the contest, compete alongside Cao Yi and many other outstanding participants, and together push the inference limits of MiniCPM-SALA! Click the image below or visit the contest website https://soar.openbmb.cn/ to learn more.