MiniCPM-V 4

MiniCPM-V Multimodal Series

MiniCPM-V 4 is a 4.1B-parameter on-device multimodal large model from ModelBest, using a SigLIP2-400M vision encoder and the MiniCPM4-3B language model (based on the LLaMA architecture). HuggingFace model ID: openbmb/MiniCPM-V-4.

Supported fine-tuning methods:

1. Official scripts: LoRA fine-tuning (finetune_lora.sh) and full-parameter fine-tuning (finetune_ds.sh)
2. LLaMA-Factory: supported via the MiniCPM-V template

Data Preparation

The fine-tuning data format is the same as for MiniCPM-V 2.6: a JSON file in which each sample contains id, image, and conversations fields.

Single-image data example:

JSON

[
  {
    "id": "0",
    "image": "path/to/image.jpg",
    "conversations": [
      {"role": "user", "content": "<image>\n描述这张图片"},
      {"role": "assistant", "content": "这是一张风景照片。"}
    ]
  }
]

Multi-image data example:

JSON

[
  {
    "id": "1",
    "image": ["path/to/img1.jpg", "path/to/img2.jpg"],
    "conversations": [
      {"role": "user", "content": "<image><image>\n比较这两张图片"},
      {"role": "assistant", "content": "第一张图片展示了...，第二张图片展示了..."}
    ]
  }
]

Fine-tuning with the Official Scripts

First clone the MiniCPM-V repository and install the dependencies:

Bash

git clone https://github.com/OpenBMB/MiniCPM-V.git
cd MiniCPM-V
pip install -r requirements.txt

LoRA Fine-tuning

Use finetune_lora.sh for LoRA fine-tuning. It uses less VRAM and suits resource-constrained scenarios.

Bash

# 编辑 finetune_lora.sh 中的数据路径和模型路径
bash finetune/finetune_lora.sh

Key parameters:

• MODEL: model path; set to openbmb/MiniCPM-V-4 or a local path
• DATA: path to the training data JSON file
• LLM_TYPE: must be set to llama (MiniCPM4-3B is based on the LLaMA architecture, not qwen)

Full-parameter Fine-tuning

Use finetune_ds.sh for full-parameter fine-tuning with DeepSpeed; this requires more VRAM.

Bash

# 编辑 finetune_ds.sh 中的数据路径和模型路径
bash finetune/finetune_ds.sh

Loading LoRA for Inference

After fine-tuning, load the LoRA adapter for inference with the following code:

Python

from peft import PeftModel
from transformers import AutoModel, AutoTokenizer

model_path = "openbmb/MiniCPM-V-4"
lora_path = "your_lora_adapter_path"

model = AutoModel.from_pretrained(model_path, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

model = PeftModel.from_pretrained(model, lora_path)
model = model.eval().cuda()

Fine-tuning with LLaMA-Factory

LLaMA-Factory already supports the MiniCPM-V series. Just select the MiniCPM-V template; LLM_TYPE likewise needs to be set to llama.

For specific configuration, refer to the MiniCPM-V examples in the official LLaMA-Factory documentation.

Related Resources

Topic	Link
Model overview	MiniCPM-V 4 overview
Code repository	OpenBMB/MiniCPM-V
Official fine-tuning scripts	finetune directory