FlagOS brings MiniCPM5 to 9 chips including ARM, covering cloud to edge on Day 0 with efficiency on par with NVIDIA native

2026-05-27

On May 25, ModelBest officially released and open-sourced MiniCPM5-1B, a new-generation on-device text foundation model. With 1B parameters, it scored 17.9 on the AA-Index, surpassing Qwen3.5-2B (16.3) and every other open-source base model under 4B, continuing the "density law" proposed by ModelBest: the intelligence density of large models doubles roughly every 3.5 months. Its Base version was pre-trained with ForgeTrain, ModelBest's self-developed AI training framework — the world's first production-grade training framework written entirely by AI. After INT4 quantization, the weights are only 0.5GB and can run on phones, browsers, and more than 90% of end devices. It natively supports mainstream inference frameworks such as vLLM, SGLang, and llama.cpp.

The community-driven FlagOS project used its unified multi-chip software stack and the vLLM-plugin-FL inference plugin to complete cross-chip adaptation and inference deployment of MiniCPM5-1B as a Day-0 release. The adaptation covers eight categories of data center AI accelerators — NVIDIA, Huawei Ascend, T-Head, Moore Threads, Hygon, Kunlunxin, MetaX, and Iluvatar CoreX — and, given MiniCPM5's focus on on-device deployment, extends Day-0 adaptation to the ARM edge platform for the first time. FlagOS supports MiniCPM5-1B on ARM at int8, bf16, and fp32 precision, achieving "one codebase, cross-architecture execution" from multiple data center AI chips to edge ARM. In terms of performance, first-token latency (TTFT) on several AI chips is lower than the NVIDIA H20 native baseline, further validating FlagOS's unified software stack for cross-chip adaptation and inference optimization.

On some chips, efficiency matches or slightly exceeds NVIDIA

Test conditions and baseline definitions:

All performance data was collected in a unified test environment: a single card, 64 concurrent requests, input lengths of 4k/16k/32k tokens, an output length of 1k tokens, and bf16 precision throughout.

The comparison baseline is defined as "NV native": the NVIDIA H20 native baseline refers to MiniCPM5-1B running on NVIDIA H20 through the native vLLM/CUDA path. On top of that, we include "NVIDIA H20 + FlagOS" as a reference to show the effect of FlagOS optimization.

Results for MiniCPM5-1B on other non-NVIDIA chips are based on FlagOS.

Multiple AI chip platforms match or beat the NVIDIA H20 native baseline on long-context TTFT

01 First-token latency (TTFT) comparison

TTFT (Time To First Token) measures the latency from receiving a request to generating the first output token; lower is better. The test results are as follows:

T-Head Zhenwu 810E: TTFT is significantly lower than NV native at every context length (4k/16k/32k), and the advantage grows as context length increases.

Moore Threads, Hygon, and MetaX: TTFT is also lower than NV native across all test scenarios (that is, faster first-token response).

NVIDIA H20 + FlagOS: thanks to kernel optimizations in FlagGems and FlagTree, TTFT beats NV native at all context lengths.

It is worth emphasizing that the comparison between "various AI chips + FlagOS" and "NV native" compares chip-plus-software-stack combinations, not pure hardware capability on equal terms. FlagOS's deep adaptation across multiple AI chips is the key factor behind the TTFT advantage.

Figure 1. First-token latency of MiniCPM5-1B on different AI chips (lower is better)

Note: because chips differ substantially in hardware compute (TFLOPS) and memory bandwidth, these figures do not represent a performance comparison between the chips themselves.

02 TTFT efficiency per unit of compute

Because peak compute and bandwidth configurations vary widely across chips, comparing absolute TTFT alone does not fairly reflect compute utilization. We therefore also calculated TTFT efficiency per unit of compute, normalized by each chip's peak TFLOPS — 1/(TFLOPS * TTFT) — to measure first-token latency more fairly at equal units of compute.

T-Head Zhenwu 810E: TTFT efficiency per unit of compute is clearly better than NV native at every context length.

MetaX and Moore Threads: TTFT efficiency per unit of compute exceeds NV native in the 16k and 32k long-context scenarios.

NVIDIA H20 + FlagOS: better than NV native at 4k and 16k, and on par with NV native at 32k.

For other platforms, we will keep optimizing the FlagOS stack to improve their compute efficiency.

Figure 2. TTFT efficiency per unit of compute for FlagOS-based MiniCPM5-1B on different AI chips (higher is better)

02 T-Head inference throughput approaches NVIDIA's efficiency at equivalent compute

TTFT mainly affects the interactive experience, while inference throughput (total tokens/s) determines how many requests a system can handle concurrently. They are performance metrics on different dimensions. Total inference throughput is typically determined jointly by a chip's compute (TFLOPS), memory bandwidth, and other important factors.

On throughput efficiency: for this adaptation we used the FlagGems + FlagTree + vLLM-plugin-FL stack on the T-Head Zhenwu 810E. Measured results show that a single Zhenwu 810E card delivers total token throughput above 26,000 tokens/s.

Because MiniCPM5-1B has a small parameter count and KV cache access is a significant burden during long-context decoding, the overall workload is more memory-bound. To compare efficiency fairly across compute chips, and after accounting for the differences between the Zhenwu 810E and NVIDIA H20 in peak compute (TFLOPS) and memory bandwidth, we normalized to equivalent effective compute. On that basis, the Zhenwu 810E's total token throughput in long-sequence scenarios such as 16K:1K and 32K:1 has reached 93.3% and 95.3% of the H20's token compute efficiency per unit of compute, respectively.

ARM on-device support: extending MiniCPM5-1B from GPUs to massive numbers of end devices

ARM architecture is everywhere — phones, AI PCs, robots, in-vehicle terminals, edge gateways, and all kinds of embedded devices — making it one of the largest and broadest compute platforms for on-device AI deployment. As small, high-density models like MiniCPM5-1B advance, more inference workloads can move from cloud GPUs down to end devices for lower latency, lower cost, better privacy protection, and offline availability. Adapting the model to ARM is therefore more than a hardware migration test; it is an important exploration by FlagOS toward on-device AI and inclusive AI compute.

FlagOS's ARM board support is a differentiating capability of this MiniCPM5-1B multi-chip effort. In the past, high-performance large-model inference optimization focused mainly on GPUs, and moving a model from GPU to ARM usually required re-adapting the inference backend, kernel implementations, and precision paths. With the FlagTree + FlagGems combination, FlagOS extends MiniCPM5-1B inference from GPUs to ARM boards, supports int8, bf16, fp32, and other precision paths, and explores an on-device deployment approach of "one model, multiple architectures, unified adaptation."

Deep ARMv9 optimization: FlagTree adds an Armv9 CPU compilation target (FlagTree-CPU) with Triton-CPU as the underlying backend, fully using SVE2 / NEON vectorization to improve general kernel throughput; it accelerates INT8 GEMM with i8mm, optimizes INT8 GEMV decode with dotprod extensions (SDOT / UDOT), and optimizes BF16 matmul with BF16 instruction extensions. The compilation pipeline enables Arm target features such as +dotprod / +bf16 / +i8mm so the CPU backend fully matches Armv9 hardware capabilities.

FlagGems ARM backend: provides ARM-optimized kernel implementations for key large-model inference modules such as matrix computation, attention, normalization, dynamic quantization, and decode post-processing. Developers can get kernel-level acceleration through the FlagOS stack without changing model code.

Triton-TLE (Triton Language Extensions) support for ARM instructions: FlagOS uses Triton-TLE to strengthen Triton's expressiveness and optimization on the ARM CPU backend, optimizing the runtime and core compute paths with NEON / SVE2 instructions, and supporting hot paths in large-model inference such as attention decode, feed-forward network fusion, normalization, gated state updates, and low-precision quantized linear computation.

Developer quick-start guide: getting and deploying multi-chip MiniCPM5-1B

The FlagOS stack provides out-of-the-box multi-chip versions of MiniCPM5-1B. With support from the vLLM-plugin-FL multi-chip inference plugin, the unified kernel library FlagGems, and the unified compiler FlagTree, chips from NVIDIA, Huawei Ascend, T-Head, Moore Threads, and others have completed cross-chip adaptation and validation for MiniCPM5-1B. FlagOS offers a unified deployment path based on vLLM-plugin-FL. To install and deploy from source, see the official one-stop developer documentation below, which includes detailed code examples and instructions:

➤ GitHub

🔗 https://github.com/flagos-ai/vllm-plugin-FL/blob/main/README.md

➤ GitCode

🔗 https://gitcode.com/flagos-ai/vllm-plugin-FL/blob/main/README.md

Option 1: Install from FlagOS source

Install and deploy

## 1. Install vLLM v0.20.2pip install vllm==0.20.2

2. Install vllm-plugin-FLgit clone https://github.com/flagos-ai/vllm-plugin-FLcd vllm-plugin-FLpip install --no-build-isolation -e .

3. Install the FlagGems kernel librarygit clone https://github.com/flagos-ai/FlagGemscd FlagGems && git checkout v5.0.0pip install --no-build-isolation -e .

4. (Optional) Install the FlagTree unified compilerpython3 -m pip uninstall -y tritonpython3 -m pip install flagtree===0.5.0 --index-url=https://resource.flagos.net/repository/flagos-pypi-hosted/simple

## 5. (Optional) Install the FlagCX unified communication library# See https://github.com/flagos-ai/FlagCX

Run inference

from vllm import LLM, SamplingParams

prompts = ["Hello, please introduce yourself."]llm = LLM( model="/path/to/MiniCPM5-1B-FlagOS", max_num_batched_tokens=16384, max_num_seqs=2048)sampling_params = SamplingParams(max_tokens=512, temperature=0.0)outputs = llm.generate(prompts, sampling_params)for output in outputs: print(f"Prompt: {output.prompt!r}") print(f"Generated: {output.outputs[0].text!r}")

Option 2: Download model images directly

Users can directly pull the migrated model files, code, and images published on FlagRelease. Below are model versions for several AI chips after migration and adaptation, ready to use out of the box with no migration required.

ModelScope

Hugging Face

A polished developer experience: multi-chip at release plus zero code changes

vLLM-plugin-FL: the core path to unified multi-chip inference

Cross-chip adaptation for MiniCPM5-1B relies on vLLM-plugin-FL as its core technical path for multi-chip inference deployment. vLLM-plugin-FL is a dedicated plugin that FlagOS built for the vLLM inference serving framework. Developed on the FlagOS unified multi-chip backend, it extends MiniCPM5-1B inference to NVIDIA, Huawei Ascend, T-Head, Moore Threads, Hygon, and other GPU chips without changing vLLM's native interfaces or users' habits at all.

Compatible with native vLLM interfaces: developers can load models and deploy inference services with the standard vLLM API, with no new deployment tool to learn and no existing code to modify.

Automatic multi-chip backend adaptation: vLLM-plugin-FL has built-in backend adaptation logic for different chips. Developers only specify the model path, and the plugin handles the underlying chip differences automatically.

Core capabilities align with the native version

Validated on four benchmarks — hellaswag, truthfulqa, winogrande, and commonsense_qa — the FlagOS-adapted MiniCPM5-1B matches the NVIDIA H20 native version's inference accuracy on most chips and benchmarks.

Minimal deployment: ready out of the box

FlagOS integrates the kernel library, compiler, and other components internally. Lower-level optimizations take effect automatically when developers load the model, with no manual FlagOS initialization code. FlagRelease provides multi-chip MiniCPM5-1B-FlagOS model files directly, and with vLLM-plugin-FL to start the inference service, going from download to running takes just a few commands.

The foundation: four FlagOS technologies behind MiniCPM5-1B's rapid cross-chip adaptation

Cross-chip adaptation for MiniCPM5-1B and other models relies on the full-stack capabilities of the FlagOS 2.0 unified multi-chip system software stack. From the kernel layer and compiler layer to the framework and tool layers, the entire chain supports large-model cross-chip adaptation.

High-performance kernel library FlagGems: deep adaptation of core kernels to unlock hardware compute

FlagGems, FlagOS's core high-performance general-purpose large-model kernel library, is implemented in Triton and already has more than 500 commonly used large-model kernels. It covers key compute modules in the MiniCPM5-1B inference path, such as attention computation and RMSNorm, and natively supports nearly 20 AI chips from NVIDIA, Moore Threads, MetaX, Tsingmicro, Iluvatar, and others. In this test, enabling just 13 FlagGems kernels was enough to push throughput on NVIDIA H20 to 2.0% above native CUDA.

Unified AI compiler FlagTree: write once, compile for many chips

FlagTree is FlagOS's unified compiler for multiple AI chip backends. Deeply customized from Triton, it compiles core large-model kernels into instructions that more than a dozen AI chip backends can recognize, including NVIDIA, Huawei Ascend, T-Head, Hygon, and Moore Threads. This eliminates the fragmented compiler ecosystems across chips and greatly lowers the development cost of adapting kernels across chips.

Cross-chip model migration and release tool FlagRelease: semi-automated migration and version release

Leveraging the full FlagOS stack, FlagRelease has completed model migration, accuracy alignment, and version release for MiniCPM5-1B and other models across multiple chips, covering open-source community platforms such as Hugging Face and ModelScope. Developers can download and use them directly without migrating anything themselves. As of publication, FlagRelease has released cross-chip adapted versions covering 10+ chip vendors, 12+ hardware models, and 70+ open-source model instances.

Unified multi-chip plugin vLLM-plugin-FL: compatible with native usage habits

vLLM-plugin-FL is the dedicated plugin FlagOS built for vLLM. Developed on the FlagOS unified multi-chip backend, it delivers multi-chip inference deployment without changing vLLM's native interfaces or user habits. vLLM-plugin-FL now supports chips from Tsingmicro, Moore Threads, Hygon, MetaX, T-Head, Iluvatar CoreX, Kunlunxin, Huawei, NVIDIA, and others.

Open-source co-building: FlagOS remains developers' cross-chip adaptation backstop

Today, "heterogeneous compute collaboration and inclusive large-model deployment" have become central topics in the global open-source developer community. Breaking down hardware ecosystem silos and running large models efficiently and cheaply on different compute platforms is a core demand for countless developers. From day one, FlagOS has built open source and community co-creation into its DNA. With developers at the center, its fully open-source unified system software stack reduces the complex "M×N" hardware adaptation problem to "M+N", making it every developer's most reliable backstop for cross-chip adaptation.

A fully open-source stack, with technical control in developers' hands

FlagOS now has a complete open-source technology system. All core components are open-sourced on GitHub, alongside adaptation solutions and best practices for dozens of the latest mainstream foundation models and more than ten AI chips. Developers can obtain and deeply customize them freely:

Four core technology libraries: the FlagGems general-purpose large-model kernel library, the FlagTree unified AI compiler, the FlagScale training and inference parallelism framework, and the FlagCX unified communication library, covering the full chain of kernel development, compilation optimization, parallel computing, and cross-chip communication.

Three open-source tool platforms: the FlagRelease automatic large-model migration and release platform, the KernelGen automatic kernel generation tool, and the FlagPerf multi-chip evaluation tool, providing a one-stop toolchain from model adaptation and performance evaluation to engineering deployment.

A full-scenario extension ecosystem: framework enhancements such as vLLM-plugin-FL, Megatron-LM-FL, and TransformerEngine-FL, plus the FlagOS-Robo embodied AI toolkit, covering training, inference, and application scenarios for large models.

Many paths to contribute, open to developers at every level

We have designed low-barrier, multi-path ways to contribute for developers across technical directions and experience levels. Whether you are new to AI development or a veteran systems software expert, you can find your place in the FlagOS community.

Beginner-friendly participation: file issues in the relevant repositories to report bugs or suggest improvements, help complete documentation, write introductory tutorials and best practices, or join community technical discussions and share your experience — start your open-source journey with zero barrier. (Community docs: https://docs.flagos.io/en/latest/)

Deep technical co-building: developers can work directly on FlagGems kernel development and optimization (new kernels / performance tuning / new chip backend support), improvements to the KernelGen kernel generation pipeline, FlagTree compiler backend extensions, and other core modules, driving the technology forward alongside core community developers.

Ecosystem tool contributions: developers can build AI agent skills for domestic chips on top of FlagOS Skills, helping more developers complete chip adaptation, model deployment, and other operations through natural language.