MiniCPM4.1 is an efficient large language model for edge devices, optimized across architecture, training data, algorithms, and inference systems.
Technical highlights
：Using a trainable sparse attention mechanism, when processing 128K long text, each token only needs to calculate correlation with less than 5% of the tokens, significantly reducing the computational overhead of long text. As shown in the RULER-NIAH stress test on the right, full retrieval accuracy is maintained in the longest context of 128K and at each depth position.
：Model Wind Tunnel 2.0 introduces an expanded prediction method for downstream task performance to achieve more accurate training configuration search; BitCPM extremely compresses the model parameter quantization bit width to three values, achieving about 90% bit width reduction; and combines FP8 low-precision calculation and multi-token prediction training strategies to further improve training efficiency.
：UltraClean builds an iterative data cleaning strategy based on efficient data verification, and open-sources the high-quality Chinese and English pre-training data set UltraFinweb; UltraChat v2 builds a large-scale high-quality supervised fine-tuning data set, covering multi-dimensional data such as knowledge-intensive, reasoning-intensive, instruction following, long text understanding, and tool invocation.
：CPM.cu's lightweight and efficient CUDA inference framework integrates sparse attention, model quantification and speculative sampling technologies to achieve efficient pre-filling and decoding; the ArkInfer cross-platform deployment system supports efficient deployment of multiple back-end environments and provides flexible cross-platform adaptation capabilities.