MiniCPM4.1 & MiniCPM4

MiniCPM Text Series

Choose Transformers, vLLM, SGLang or CPM.cu based on your runtime environment, and learn about dense and sparse inference, hybrid thinking and speculative sampling as needed. MiniCPM4.1 also provides entry points for llama.cpp and Ollama — please note which model version each section applies to.

Model Inference

You can run inference with Huggingface Transformers, vLLM, SGLang and CPM.cu.

If inference efficiency is your priority, try CPM.cu first.

MiniCPM4 / MiniCPM4.1 support two modes, dense inference and sparse inference. vLLM and SGLang currently support only the dense inference mode. To use sparse inference, please use Huggingface Transformers or CPM.cu.

• Dense attention inference: vLLM, SGLang, Huggingface Transformers
• Sparse attention inference: Huggingface Transformers, CPM.cu

Hybrid Thinking (MiniCPM4.1)

MiniCPM4.1 supports hybrid thinking, which can be used for both deep thinking and non-thinking modes. Users can enable thinking mode by setting enable_thinking=True, and enable non-thinking mode by setting enable_thinking=False. Likewise, users can append /no_think to the end of a query to enable non-thinking mode. If no special token is added, or /think is appended to the end of the query, the model enables thinking mode.

Python

# Enable reasoning mode
prompt_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True
)
# Enable non-reasoning mode
prompt_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)

Huggingface Transformers

Dense Attention Inference

Python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)

path = 'openbmb/MiniCPM4.1-8B'
device = "cuda"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map=device, trust_remote_code=True)

# User can directly use the chat interface
# responds, history = model.chat(tokenizer, "Write an article about Artificial Intelligence.", temperature=0.7, top_p=0.7)
# print(responds)

# User can also use the generate interface
messages = [
    {"role": "user", "content": "Write an article about Artificial Intelligence."},
]
prompt_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
model_inputs = tokenizer([prompt_text], return_tensors="pt").to(device)

model_outputs = model.generate(
    **model_inputs,
    max_new_tokens=32768,
    top_p=0.95,
    temperature=0.6
)
output_token_ids = [
    model_outputs[i][len(model_inputs[i]):] for i in range(len(model_inputs['input_ids']))
]

responses = tokenizer.batch_decode(output_token_ids, skip_special_tokens=True)[0]
print(responses)

Sparse Attention Inference

This model supports the sparse attention mechanism InfLLM v2, which handles long-sequence inference efficiently. To enable it, first install the dependency library infllmv2_cuda_impl.

Run the following commands to install it:

Bash

git clone -b feature_infer https://github.com/OpenBMB/infllmv2_cuda_impl.git
cd infllmv2_cuda_impl
git submodule update --init --recursive
pip install -e . # or python setup.py install 

To enable InfLLM v2, add a sparse_config field to the config.json configuration file:

JSON

{
    ...,
    "sparse_config": {
        "kernel_size": 32,
        "kernel_stride": 16,
        "init_blocks": 1,
        "block_size": 64,
        "window_size": 2048,
        "topk": 64,
        "use_nope": false,
        "dense_len": 8192
    }
}

Configuration parameters

Parameter	Default	Description
kernel_size	32	Size of the semantic kernel
kernel_stride	16	Stride between adjacent semantic kernels
init_blocks	1	Number of initial blocks each query token attends to, ensuring the model can attend to the beginning of the sequence
block_size	64	Block size of the key-value blocks
window_size	2048	Size of the local sliding window
topk	64	Each token computes attention only with the top-k most relevant key-value blocks
use_nope	false	Whether to use the NOPE technique in block selection to improve performance
dense_len	8192	When the token length is below this threshold, automatically switches to standard attention; when set to -1, sparse attention is always used

Length Extension

MiniCPM4.1 natively supports a context length of 65,536 tokens. If the total conversation length (input + output) far exceeds this limit, we recommend extending the context with RoPE scaling. By adjusting the LongRoPE factors, the model can stably support an ultra-long context of 131,072 tokens.

How to modify: adjust the parameters of the rope_scaling field in the config.json file.

JSON

{
    ...,
    "rope_scaling": {
        "rope_type": "longrope", 
        "long_factor": [0.9982316082870437, 1.033048153422584, 1.0749920956484724, 1.1255096879436193, 1.1863348602111476, 1.259543828902579, 1.3476188888731149, 1.4535223827776373, 1.5807816745852985, 1.7335856049489526, 1.9168922912975785, 2.1365471404135326, 2.3994084200118646, 2.713475511863602, 3.0880118452194134, 3.533650295140154, 4.062463396503134, 4.687974098908333, 5.425075306704039, 6.289818967956352, 7.29902962722721, 8.6357018163639, 10.210822723989212, 12.053807765671676, 14.193944598909404, 16.65780676784363, 19.463620727694074, 22.628311203524586, 26.150106147261315, 30.02526691405111, 34.23183327975347, 38.73811934094828, 43.502489489729555, 48.47627117965394, 53.61139491762471, 58.857366522037935, 64.16798299215064, 69.51359464319125, 74.86555458220285, 80.21497790341579, 85.55322183307433, 90.89611806932027, 96.26245306514224, 101.68269304046481, 107.18619510219668, 112.82253283014026, 118.63764063163615, 119.88866203644656, 120.9462882391725, 121.837565139014, 122.58663780572562, 123.2147719894291, 123.74049454862576, 124.17980424685767, 124.54641761955492, 124.85202548028222, 125.10654406389756, 125.31835105170659, 125.49450117164764, 125.64091910903052, 125.76256945356558, 125.86360463815589, 125.94749252260765, 126.01712561287873],
        "short_factor": [0.9982316082870437, 1.033048153422584, 1.0749920956484724, 1.1255096879436193, 1.1863348602111476, 1.259543828902579, 1.3476188888731149, 1.4535223827776373, 1.5807816745852985, 1.7335856049489526, 1.9168922912975785, 2.1365471404135326, 2.3994084200118646, 2.713475511863602, 3.0880118452194134, 3.533650295140154, 4.062463396503134, 4.687974098908333, 5.425075306704039, 6.289818967956352, 7.29902962722721, 8.6357018163639, 10.210822723989212, 12.053807765671676, 14.193944598909404, 16.65780676784363, 19.463620727694074, 22.628311203524586, 26.150106147261315, 30.02526691405111, 34.23183327975347, 38.73811934094828, 43.502489489729555, 48.47627117965394, 53.61139491762471, 58.857366522037935, 64.16798299215064, 69.51359464319125, 74.86555458220285, 80.21497790341579, 85.55322183307433, 90.89611806932027, 96.26245306514224, 101.68269304046481, 107.18619510219668, 112.82253283014026, 118.63764063163615, 119.88866203644656, 120.9462882391725, 121.837565139014, 122.58663780572562, 123.2147719894291, 123.74049454862576, 124.17980424685767, 124.54641761955492, 124.85202548028222, 125.10654406389756, 125.31835105170659, 125.49450117164764, 125.64091910903052, 125.76256945356558, 125.86360463815589, 125.94749252260765, 126.01712561287873],
        "original_max_position_embeddings": 65536
    }
}

---

vLLM

You can accelerate generation with speculative sampling, or deploy the model in standard mode.

Speculative Sampling

To use speculative sampling for accelerated inference with vLLM:

1. Download the MiniCPM4.1 draft model

Bash

cd /your_path
git clone https://huggingface.co/openbmb/MiniCPM4.1-8B-Eagle3

1. Install EAGLE3-compatible vLLM

The vLLM PR for EAGLE3 has been submitted. For now, please install from our repository:

Bash

git clone https://github.com/LDLINGLINGLING/vllm.git
cd vllm 
pip install -e .

1. Start the vLLM service with speculative sampling

Make sure to update the model path in speculative-config to the downloaded MiniCPM4_1-8B-Eagle3-bf16 folder:

Bash

VLLM_USE_V1=1 \
vllm serve openbmb/MiniCPM4.1-8B \
--seed 42 \
--trust-remote-code \
--speculative-config '{
  "model": "your/path/MiniCPM4_1-8B-Eagle3-bf16",
  "num_speculative_tokens": 3,
  "method": "eagle3",
  "draft_tensor_parallel_size": 1
}'

1. Client usage example

Client usage is the same for standard decoding and speculative sampling:

Python

import openai

client = openai.Client(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="openbmb/MiniCPM4.1-8B",
    messages=[
        {"role": "user", "content": "Write an article about Artificial Intelligence."},
    ],
    temperature=0.6,
    max_tokens=32768,
    extra_body=dict(add_special_tokens=True),  # 确保在聊天模板中加入特殊符号
    
)

print(response.choices[0].message.content)

Configuration parameters

Parameter	Default / Example	Description
VLLM_USE_V1	1	Enables the vLLM v1 API
--speculative-config	JSON config	Speculative sampling configuration, specifying the draft model, number of speculative tokens, sampling method and other parameters
model	Draft model path	Path of the draft model used for speculative sampling
num_speculative_tokens	3	Number of tokens speculatively generated each time
method	eagle3	Speculative sampling method
draft_tensor_parallel_size	1	Tensor parallel size of the draft model
--seed	Random seed	Sets the random seed to ensure reproducible results
--trust-remote-code	None	Allows execution of remote code to support custom model architectures or implementations

Standard Inference (without Speculative Sampling)

For now, you need to install the latest version of vLLM.

Bash

pip install -U vllm \
    --pre \
    --extra-index-url https://wheels.vllm.ai/nightly

Then you can run MiniCPM4.1-8B inference with vLLM:

Python

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_name = "openbmb/MiniCPM4.1-8B"
prompt = [{"role": "user", "content": "Write an article about Artificial Intelligence."}]

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)

llm = LLM(
    model=model_name,
    trust_remote_code=True,
    max_num_batched_tokens=65536,
    dtype="bfloat16", 
    gpu_memory_utilization=0.8, 
)
sampling_params = SamplingParams(top_p=0.95, temperature=0.6, max_tokens=32768)

outputs = llm.generate(prompts=input_text, sampling_params=sampling_params)

print(outputs[0].outputs[0].text)

You can also start an inference service with the following command:

Bash

vllm serve openbmb/MiniCPM4.1-8B --trust-remote-code

Note: In vLLM's chat API, add_special_tokens defaults to False. This means important special tokens — such as the beginning-of-sequence token (BOS token) — are not added automatically. To make sure the input prompt is correctly formatted for the model, we recommend explicitly setting extra_body={"add_special_tokens": True}.

Then you can use the chat interface with the following code:

Python

import openai

client = openai.Client(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="openbmb/MiniCPM4.1-8B",
    messages=[
        {"role": "user", "content": "Write an article about Artificial Intelligence."},
    ],
    temperature=0.6,
    max_tokens=32768,
    extra_body=dict(add_special_tokens=True),  # 确保在聊天模板中加入特殊符号
)

print(response.choices[0].message.content)

---

SGLang

You can accelerate generation with speculative sampling, or deploy the model in standard mode.

Speculative Sampling

To use speculative sampling for accelerated inference:

1. Download the MiniCPM4.1 draft model

Bash

cd /your_path
git clone https://huggingface.co/openbmb/MiniCPM4.1-8B-Eagle3

1. Install EAGLE3-compatible SGLang

The adaptation PR for EAGLE3 has been submitted. For now, please install from our repository:

Bash

git clone https://github.com/LDLINGLINGLING/sglang.git
cd sglang
pip install -e .

1. Start the SGLang service with speculative sampling

Bash

python -m sglang.launch_server \
  --model-path "openbmb/MiniCPM4.1-8B" \
  --host "127.0.0.1" \
  --port 30002 \
  --mem-fraction-static 0.9 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path "your/path/MiniCPM4_1-8B-Eagle3-bf16" \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 32 \
  --temperature 0.7

1. Client usage example

Client usage is the same for standard decoding and speculative sampling:

Python

import openai

client = openai.Client(base_url=f"http://localhost:30002/v1", api_key="None")

response = client.chat.completions.create(
    model="openbmb/MiniCPM4.1-8B",
    messages=[
        {"role": "user", "content": "Write an article about Artificial Intelligence."},
    ],
    temperature=0.6,
    max_tokens=32768,
)

print(response.choices[0].message.content)

Note: Make sure to update the port number in the client code to match the server port (30002 in the speculative sampling example).

Configuration parameters

Parameter	Default / Example	Description
--speculative-algorithm	EAGLE3	Enables the EAGLE3 speculative sampling algorithm.
--speculative-draft-model-path	Draft model path	Path of the draft model used for speculative sampling.
--speculative-num-steps	3	Number of speculative steps, i.e. how many steps each round of speculative sampling looks ahead.
--speculative-eagle-topk	1	EAGLE top-k parameter, controlling the candidate range of draft tokens.
--speculative-num-draft-tokens	32	Number of draft tokens generated by the draft model.
--mem-fraction-static	0.9	Fraction of VRAM statically allocated, controlling how much GPU memory the inference service pre-allocates.

Standard Inference (without Speculative Sampling)

For now, you need to install our fork of SGLang.

Bash

git clone -b openbmb https://github.com/OpenBMB/sglang.git
cd sglang

pip install --upgrade pip
pip install -e "python[all]"

You can start an inference service with the following command:

Bash

python -m sglang.launch_server --model openbmb/MiniCPM4.1-8B --trust-remote-code --port 30000 --chat-template chatml

Then you can use the chat interface with the following code:

Python

import openai

client = openai.Client(base_url=f"http://localhost:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="openbmb/MiniCPM4.1-8B",
    messages=[
        {"role": "user", "content": "Write an article about Artificial Intelligence."},
    ],
    temperature=0.6,
    max_tokens=32768,
)

print(response.choices[0].message.content)

---

CPM.cu

We recommend using CPM.cu for inference with MiniCPM4 and MiniCPM4.1. CPM.cu is a CUDA inference framework developed by ModelBest that integrates efficient sparsity, speculative sampling, quantization and other techniques, fully realizing the efficiency advantages of MiniCPM4 and MiniCPM4.1.

You can install CPM.cu and run inference with the following script:

Bash

git clone https://github.com/OpenBMB/CPM.cu.git --recursive
cd CPM.cu
python3 setup.py install

You can run inference and check the model's speed with the following commands.

Bash

python3 tests/long_prompt_gen.py # 生成 prompt.txt
python3 tests/test_generate.py --prompt-file prompt.txt

You can use EAGLE3 for speculative inference with the following command.

Bash

python3 -m cpmcu.cli \
    --model-path $BASE_MODEL_PATH \
    --draft-model-path $EAGLE3_DRAFT_MODEL_PATH \
    --prompt-text "Tell me about Tsinghua University" \
    --use-eagle3 true

For more details about CPM.cu, see the CPM.cu repository.

---

llama.cpp

You can download the GGUF-format MiniCPM4.1-8B model from huggingface and run inference with llama.cpp.

Bash

# case 1: main-cli
./build/bin/llama-cli -m MiniCPM4.1-8B-Q4_K_M.gguf -p "Write an article about Artificial 
Intelligence." -n 1500

# case 2: server
## launch server
./build/bin/llama-server -m MiniCPM4.1-8B-Q4_K_M.gguf --host 127.0.0.1 --port 8080 -c 
4096 -fa on &

## send request
curl -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Write an article about Artificial 
    Intelligence."}],
    "max_tokens": 1500
  }'

Ollama

Go to the model library to download the model. After installing Ollama, you can use MiniCPM4.1 with the following command:

Bash

ollama run openbmb/minicpm4.1