ModelBest Lookahead Expert Scheduling (LES): Running a 35B MoE On-Device in Real Time, Losslessly
Figure 1: The MiniCPM-V 4.7 series leads on both inference speed and evaluation scores
ModelBest Lookahead Expert Scheduling (LES) is the core technology behind MiniCPM-V 4.7's efficient on-device operation. It is the first to run a 35B-A3B-scale MoE model losslessly in real time on-device, measured at over 20 tokens/s decoding on an iPhone 18 Pro Max, raising the scale of models that can run in real time on-device by an order of magnitude. This article describes the core design of the LES inference system and its optimizations at the scheduling, storage, and compute layers.
Goals and challenges
Our goal is to run models losslessly in real time on-device. Real time means decoding above 8 tokens/s; lossless means fully preserving the original model's network structure and inference mechanism — no reduction in the number of experts, activated experts, or network layers, and no fixing expert selection via prompts or degrading the MoE model into a dense model.
To meet this goal, the system design must satisfy three constraints: first, fully execute all experts selected by the real router, without reducing weight reads by substituting experts or cutting the number of activated experts; second, avoid training an additional expert-prediction module, so no extra training cost is introduced; third, keep runtime memory usage within what mainstream on-device devices can support.
Under these constraints, we need to solve three core problems.
Weights don't fit: model size exceeds available memory
MiniCPM-V 4.7 35B-A3B has about 35 billion total parameters and activates about 3 billion per token. At 4 bits per parameter, the weights alone need about 17.5 GB, not counting quantization metadata, KV Cache, and runtime buffers. On an iPhone 18 Pro Max, for example, 12 GB of unified memory is shared by the system, CPU, GPU, and apps, so the full model weights cannot stay resident in memory.
The sparse activation of MoE offers another angle: each layer only needs to execute a small number of experts, so expert weights can stay in flash storage, be loaded into memory when needed, and be released after computation. That way, the memory needed to run the model does not have to scale with the model's full parameter count.
But weight offloading does not eliminate the cost — it turns accesses that used to happen in memory into flash reads.
Reads block compute: expert loading sits on the critical path
Every MoE layer must first have the router select experts, then load the corresponding weights, and finally execute the computation. If these three steps run serially, computation has to wait for weights to arrive.
At an effective read bandwidth of 1 GB/s, reading 1.29 MB of expert weights takes about 1.29 ms. Every token passes through multiple layers, and each layer needs to load several experts, so the read wait accumulates layer by layer.
Prefetching can overlap weight loading with current computation, but there is a key difficulty: the real input to the next layer's router depends on the current layer's unfinished computation, so the next layer's true expert selection cannot be obtained directly in advance.
The system needs to predict likely experts before the real routing result exists and to arrange the order of weight reads and computation. The former determines which weights are worth loading early; the latter determines how to hide read latency as much as possible.
Low compute density per token: limited useful output per round
Conventional autoregressive decoding processes only one new token per step, so the amount of computation is relatively limited and hard-pressed to fully cover flash read overhead.
Even if prefetching reduces the wait, the model still advances layer by layer, and every generated token bears a round of decoding computation and weight-access cost.
So the system also needs to raise the useful output per round — using speculative decoding so that one target-model execution can accept multiple candidate tokens and generate multiple output tokens at once when verification succeeds, thereby lowering the execution cost amortized over each output token.
LES overview
LES is ModelBest's inference optimization solution for on-device deployment of MiniCPM-V 4.7 35B-A3B, supporting on-device platforms including iOS, Android, and HarmonyOS.
LES addresses the problems above at three layers:
Scheduling layer: predict the experts likely used by the next layer ahead of time, overlap weight reads with computation, and prioritize experts whose weights are already ready.
Storage layer: reduce scattered reads through a contiguous expert layout, and cut weight size and transfer volume through mixed-precision quantization.
Compute layer: batch-verify candidate tokens with DSpark to increase the useful output of each target-model execution round.
The three layers work together: the scheduling layer hides as much remaining read latency as possible; the storage layer reduces read volume and improves read efficiency so more weights arrive within the compute window; the compute layer further lowers the cost amortized over each output token.
Scheduling
Predicting the next layer's experts from the previous layer's input
Figure 2: Lookahead expert scheduling strategy
Among existing expert prefetching schemes, one class builds lookup tables from offline statistics of expert selection patterns, then queries them online to load weights early. Such methods depend on the consistency between the statistics and the actual inference distribution; when the model or task distribution changes, recalibration may be needed.
Another class reuses the model's own router to predict the experts of adjacent layers, an idea also used for moving experts between GPU VRAM and host memory. But phone flash offloading faces different system constraints: bandwidth between memory and VRAM is usually far higher than the effective read bandwidth of phone flash, and the headroom of on-device unified memory is more limited. The extra read from a wrong prefetch, or the wait from a miss, can directly affect decoding speed. So on-device MoE needs not only expert prediction but also a complete read-and-compute scheduling mechanism built around the prediction.
LES reuses the model's own router for prediction and needs no additional trained predictor. The basis comes from the structure of residual networks: for two adjacent layers, the later layer's input contains the previous layer's input plus the previous layer's computed update, so the representations of adjacent layers are correlated. LES exploits this property by feeding the current layer's router input into the next layer's router early, obtaining the next layer's candidate experts while the current layer's experts are still computing, and immediately issuing flash reads.
It is important to stress that the prediction results are only used to decide which weights to load early. After the next layer's real router output is produced, the real routing still determines the experts ultimately executed. A wrong prediction does not change the set of experts actually involved in computation, nor does it change how the real routing weights are used.
LES thus separates expert prediction from actual routing: prediction is responsible for buying loading time, and the real router is responsible for guaranteeing computational correctness. Reusing the existing router weights also avoids training a separate prediction module for every model.
In measurements, the recall of candidate experts is about 80%. Expanding the candidate set from top-k to top-2k raises recall by about 15%, but read volume and buffer usage increase accordingly. We use top-k as the default configuration, trading off prediction coverage against extra read overhead.
Ready experts compute first
Prediction can only increase the probability that the right weights are loaded early; it cannot guarantee that all experts selected by the real router are ready. So even with lookahead prefetching, some experts may still need to wait for flash reads.
For example, suppose experts A, B, and C are prefetched, but the real router ends up selecting A, C, and D. A and C are already in memory, while D is still being read from flash. If the system had to wait for D before starting computation, A and C could have computed first but would sit idle.
LES uses a ready-first execution policy: compute A and C, which are ready, while continuing to read D; once D arrives, finish the remaining computation. Although B was read early, it does not take part in this round's actual computation.
What is ultimately executed is still A, C, and D; expert selection, routing weights, and output merging all follow the real router's results. LES changes only the timing of weight preparation and the execution order of ready tasks, letting computation that might otherwise be blocked by reads start as early as possible.
Cache window: trading memory for throughput
Besides prefetching and ready-first execution, the expert cache also significantly affects performance.
Experts that have already been loaded can stay in memory for reuse by later tokens. The cache window c denotes the number of slots available for caching experts. When c = 0, experts are released after use, keeping only the buffers for experts in use and being loaded. The smaller c is, the lower the memory usage, but the higher the chance of repeated reads; the larger c is, the more cache hits, at the cost of higher memory usage.
On an M4 iPad, with the low-memory configuration of c = 0, peak inference memory is about 3 GB (including KV Cache) and decoding speed is 9.6–10.7 tokens/s; at c = 128, peak memory rises to 7.8 GB and decoding speed reaches 18 tokens/s. The estimated throughput with all experts resident in memory is 21.7 tokens/s.
By allocating cache space sensibly, it is possible to approach the inference performance of a larger cache configuration within a memory budget, and the exact window size can be adjusted according to the device's available memory.
Storage
Scheduling decides when to read weights, but the efficiency of the reads themselves matters just as much. LES optimizes expert weight access along two directions: storage layout and quantization.
Contiguous expert layout
Figure 3: Storage layout optimization
The inference engine uses the GGUF weight format. We found that the native weight layout is not suited to flash offloading at the granularity of a single expert: the gate, up, and down matrices of the same expert sit in different tensor regions, so reading one expert requires several scattered accesses.
To address this, we reorganize the storage layout of expert weights layer by layer, placing each expert's gate, up, and down matrices contiguously and storing the weights of different experts as contiguously as possible. This way, one expert corresponds to one contiguous region of the file. For multiple target experts adjacent in file position, the system can merge read requests to make fuller use of flash sequential read capability and reduce the overhead of scattered access.
This optimization does not change how experts are computed; it only changes how weights are organized in the file, making the storage layout better fit the on-demand loading execution pattern.
Mixed-precision quantization
Figure 4: Mixed-precision quantization
Contiguous layout improves read efficiency, and quantization further reduces the amount of data that must be read. In flash offloading scenarios, the benefit of quantization is not only a smaller model file but also fewer bytes transferred per expert load. The smaller the read volume, the more likely weights arrive before the current compute window ends, improving the effectiveness of lookahead prefetching.
Building on Q4_K_M quantization, we adopt a mixed-precision scheme that lowers expert weights further to Q3_K while keeping the remaining weights at Q4_K_M, calibrated with imatrix.
The average score across five benchmarks moves from 78.28 at FP16 to 78.12, a drop of only 0.16 points; the weight file shrinks from 19.7 GB at Q4_K_M to 14.44 GB, a 26.7% reduction. This shows that mixed-precision quantization can significantly reduce weight storage and transfer overhead while keeping the change in evaluation scores small, achieving a "lossless" effect.
Compute
DSpark batch verification
Even though scheduling and storage optimizations have reduced the wait for expert reads, the compute density of conventional single-token decoding remains limited. To further improve end-to-end throughput, we adapted the DSpark speculative decoding scheme (from DeepSeek and Peking University).
The basic idea is for a draft module to propose multiple candidate tokens, which the target model then verifies in one pass. When candidate tokens are accepted, one target-model execution can produce multiple output tokens, amortizing the execution cost per output token.
We adapted DSpark to llama.cpp, and the relevant PR has been merged upstream. Combined with LES's expert prefetching and storage optimizations, batch verification further improves overall on-device inference throughput.
Experimental results
LES achieves real-time decoding on multiple on-device devices. On an iPhone 18 Pro Max, measured decoding speed exceeds 20 tokens/s (Prefill is 113.6–176.1 tokens/s, varying with input length). On an M4 iPad, measured decoding speed is 18.0 tokens/s (Prefill throughput about 110 tokens/s).
To assess the contribution of each technique, we added optimizations step by step on the same iPhone with 12 GB of memory and measured decoding throughput, with the baseline reading expert weights entirely on demand from flash.
Figure 5: Ablation study
As shown in Figure 5, baseline decoding throughput is 3.38 tokens/s; adding lookahead expert scheduling raises it to 6.57 tokens/s, adding scheduling and storage optimizations raises it to 11.34 tokens/s, adding mixed-precision quantization brings it to 17.16 tokens/s, and adding DSpark batch verification reaches 20.51 tokens/s — 6.07× the baseline for the full system. This shows that the bottleneck of on-device MoE inference is not just the model's compute but the combined effect of memory capacity, flash access, and execution efficiency. Only system-level co-optimization across different bottlenecks can turn model scale into usable on-device performance.
On the M4 iPad, whole-device power during inference is 15.60–18.31 W. Subtracting 6.55 W of idle power with the screen on, inference adds about 9.05–11.76 W. Running continuously for 180 seconds at about 25°C, the device warms by about 1.5°C.
Conclusion
Figure 6: The frontier of open-source models that run losslessly in real time on-device (MMMU Val)
LES separates the MoE model's real routing from weight-loading scheduling: the real router determines the computation that must be executed, the scheduling layer prepares weights in advance and prioritizes ready tasks, the storage layer lowers read cost through contiguous layout and mixed-precision quantization, and the compute layer raises the useful output of each round through DSpark batch verification. Working together, the three layers let MiniCPM-V 4.7 reach over 20 tokens/s decoding within limited on-device memory while preserving the original model's network structure and real expert routing mechanism.
LES's design does not depend on a model structure unique to MiniCPM-V 4.7. Its core ideas rest on MoE's router mechanism, the correlation of adjacent-layer representations, and the fact that expert weights can be loaded independently. In principle, the approach can therefore extend to other MoE models.
We have open-sourced the LES implementation, including expert scheduling, weight access optimization, and the DSpark adaptation, at GitHub: tc-mb/llama.cpp-omni, hoping to provide a reusable technical foundation for on-device deployment of more MoE models.
Over the past few years, MoE has become an important architecture for large models, but as parameter counts keep growing, on-device deployment remains limited by memory capacity and weight access cost. Through system-level optimization, LES lets a model avoid keeping all expert weights resident in memory while still reading them in time to complete computation when needed, giving larger MoE models a chance to run in real time on the device in users' hands. On-device MoE thus moves from impossible to real — the starting point for on-device intelligence's next stage.
Written by: the MiniCPM-V team at the ModelBest Multimodal Lab
About the team
MiniCPM-V is a series of efficient on-device multimodal large models developed by the ModelBest Multimodal Lab. The lab is led by Yuan Yao, co-chief scientist at ModelBest, and focuses on the R&D and on-device deployment of multimodal large models.
Since its release in April 2024, the MiniCPM-V series has continued to iterate. The latest open-source release, MiniCPM-V 4.7, runs a 35B MoE model smoothly in a low-memory configuration of about 3 GB and achieves Day 0-level adaptation across multiple chips. To date, the MiniCPM series has surpassed 60 million cumulative downloads.