MiniCPM-SALA

MiniCPM Text Series

Prepare your runtime environment according to your context length and device, then choose either the Transformers or the SGLang route. The SGLang guide covers one-click installation, manual installation, and server configuration; before running it, check your CUDA, compiler, and related attention-kernel dependencies.

Hardware Requirements

MiniCPM-SALA's hybrid attention architecture significantly reduces the VRAM required for long contexts:

Context length	VRAM required	Reference device
32K	~20 GB	RTX 3090/4090
128K	~24 GB	RTX 3090/4090
1M	~24 GB	RTX 5090

A conventional full-attention 8B model needs 100+ GB of VRAM at 1M context; MiniCPM-SALA needs only 24 GB.

Model Inference

Recommended inference parameters: Temperature=0.9

Hugging Face Transformers

MiniCPM-SALA is compatible with Hugging Face Transformers. You can run inference with the following code:

Python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "openbmb/MiniCPM-SALA"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, device_map="auto")
model.eval()

prompts = ["My name is", "The capital of China is"]
with torch.no_grad():
    inputs = tokenizer(prompts, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs)
output_texts = tokenizer.batch_decode(outputs)
print(output_texts)

SGLang

Requirements:

• CUDA 12.x or later
• gcc / g++ compilers
• uv package manager (the script detects it automatically)

Option 1: One-click Installation

Bash

# 克隆仓库
git clone -b minicpm_sala https://github.com/OpenBMB/sglang.git
cd sglang

# 一键安装（自动创建虚拟环境并编译所有依赖）
bash install_minicpm_sala.sh

# 或指定 PyPI 镜像源
bash install_minicpm_sala.sh https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

The installation script automatically performs the following steps:

1. Creates the sglang_minicpm_sala_env virtual environment (Python 3.12)
2. Clones dependencies into the 3rdparty/ directory (infllmv2) and initializes submodules (sparse_kernel)
3. Installs MiniCPM-SALA (the current repository)
4. Compiles and installs infllmv2_cuda_impl
5. Compiles and installs sparse_kernel
6. Installs tilelang and flash-linear-attention

Option 2: Manual Installation

If the one-click script doesn't meet your needs, you can run the steps individually:

Bash

# 0. 确保 uv 可用
pip install uv

# 1. 创建虚拟环境
uv venv --python 3.12 sglang_minicpm_sala_env
source sglang_minicpm_sala_env/bin/activate

# 2. 安装 SGLang
uv pip install --upgrade pip setuptools wheel
uv pip install -e ./python[all]

# 3. 编译安装 CUDA 扩展
# (确保依赖已克隆到 3rdparty/ 目录)
cd 3rdparty/infllmv2_cuda_impl && python setup.py install && cd ../..
cd 3rdparty/sparse_kernel && python setup.py install && cd ../..

# 4. 安装额外依赖
uv pip install tilelang flash-linear-attention

Usage

Bash

# 激活环境
source sglang_minicpm_sala_env/bin/activate

# 启动推理服务（将 MODEL_PATH 替换为实际模型路径）
MODEL_PATH=/path/to/your/model

python3 -m sglang.launch_server \
    --model ${MODEL_PATH} \
    --trust-remote-code \
    --disable-radix-cache \
    --attention-backend minicpm_flashinfer \
    --chunked-prefill-size 8192 \
    --max-running-requests 32 \
    --skip-server-warmup \
    --port 31111 \
    --dense-as-sparse

Parameters

Parameter	Description
--trust-remote-code	Allow loading the model's bundled custom code
--disable-radix-cache	Disable the RadixAttention prefix cache
--attention-backend minicpm_flashinfer	Use the MiniCPM FlashInfer attention backend
--chunked-prefill-size 8192	Chunked prefill size
--max-running-requests 32	Maximum number of concurrent inference requests
--skip-server-warmup	Skip server warmup
--port 31111	Server port
--dense-as-sparse	Use dense-as-sparse mode