MiniCPM-o 2.6

MiniCPM-o Omnimodal Series

MiniCPM-o 2.6 is an 8B-parameter omnimodal model from ModelBest (OpenBMB). Built on a SigLIP-400M + Whisper-medium-300M + ChatTTS-200M + Qwen2.5-7B architecture, it integrates vision, speech and text capabilities, with overall performance comparable to GPT-4o-202405.

Supported fine-tuning methods: official scripts (LoRA / full-parameter), LLaMA-Factory, ms-swift (SWIFT), Align-Anything (DPO)

Data Preparation

Fine-tuning data uses JSON format, consistent with the MiniCPM-V series. Each sample contains three fields: id, image and conversations.

Single-image data format

JSON

[
  {
    "id": "0",
    "image": "path/to/image.jpg",
    "conversations": [
      {"role": "user", "content": "<image>\n描述这张图片"},
      {"role": "assistant", "content": "这是一张..."}
    ]
  }
]

Multi-image data format

JSON

[
  {
    "id": "0",
    "image": ["path/to/image1.jpg", "path/to/image2.jpg"],
    "conversations": [
      {"role": "user", "content": "<image>\n<image>\n比较这两张图片"},
      {"role": "assistant", "content": "第一张图片..."}
    ]
  }
]

Fine-tuning with the Official Scripts

After cloning the MiniCPM-o repository, set the MODEL and LLM_TYPE parameters in the fine-tuning script and start training.

LoRA fine-tuning

Bash

MODEL="openbmb/MiniCPM-o-2_6"
LLM_TYPE="qwen"

bash finetune/finetune_lora.sh

Full-parameter fine-tuning

Bash

MODEL="openbmb/MiniCPM-o-2_6"
LLM_TYPE="qwen"

bash finetune/finetune_ds.sh

LoRA Inference

After fine-tuning, load the LoRA adapter with the peft library for inference.

Python

from peft import PeftModel
from transformers import AutoModel, AutoTokenizer

model_path = "openbmb/MiniCPM-o-2_6"
lora_path = "path/to/lora_adapter"

model = AutoModel.from_pretrained(model_path, trust_remote_code=True)
lora_model = PeftModel.from_pretrained(model, lora_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# 合并权重（可选，用于部署）
merged_model = lora_model.merge_and_unload()

Fine-tuning with LLaMA-Factory

LLaMA-Factory supports the MiniCPM-o series; use the minicpm_o template.

LoRA fine-tuning

Bash

llamafactory-cli train examples/train_lora/minicpm_o_lora_sft.yaml

Full-parameter fine-tuning

Bash

llamafactory-cli train examples/train_full/minicpm_o_full_sft.yaml

VRAM Reference

VRAM (GPU memory) usage for different GPU configurations (A100 80GB):

Method	2x A100 80GB	4x A100 80GB	8x A100 80GB
LoRA	14.4 GiB	13.6 GiB	13.1 GiB
Full-parameter	16.0 GiB	15.8 GiB	15.63 GiB

Troubleshooting OOM

If you run out of GPU memory (OOM) during training, try the following in order:

1. Reduce batch_size or per_device_train_batch_size
2. Reduce the input image resolution or the maximum number of slices
3. Use LoRA instead of full-parameter fine-tuning for lower VRAM usage
4. Enable DeepSpeed ZeRO-2 or ZeRO-3 for memory optimization
5. Use gradient checkpointing to reduce activation memory
6. Add more GPUs to spread the memory load

Related Resources

Topic	Link
Model overview	MiniCPM-o 2.6 overview
Code repository	OpenBMB/MiniCPM-o
LLaMA-Factory framework	hiyouga/LLaMA-Factory
ms-swift framework	modelscope/ms-swift