MiniCPM-V 4.7

MiniCPM-V Multimodal Series

Adapt the model to your own task. ms-swift is the training path currently verified for MiniCPM-V 4.7, and the official project provides complete configurations and evaluation results for 1B and 35B-A3B; LLaMA-Factory offers a general-purpose LoRA and full-parameter flow.

Option	MiniCPM-V 4.7 support	Best for
ms-swift	Requires PR #10338, verified for both 1B / 35B-A3B	Recommended, with complete reference configs and evaluation
LLaMA-Factory	General-purpose multimodal flow	LoRA / full-parameter, config-file driven

MiniCPM-V 4.7 is officially supported from transformers>=5.18.0. When installing dependencies for any option, make sure the last installed transformers version is 5.18.0 or newer.

ms-swift

The counting task allenai/pixmo-count is used as the fine-tuning example. The target output format differs slightly between the two sizes:

• 1B: first lists the position of each target object point in x y coordinates (for example, <point>573 489</point>), then outputs the final count.
• 35B-A3B: outputs the final count directly as a single integer, such as 0, 3, or 10.

1B: Environment Setup

Bash

conda create -n "MiniCPM-V-4.7-1B-Counting" python=3.10 -y
conda activate "MiniCPM-V-4.7-1B-Counting"

pip install torch==2.8.0 torchvision==0.23.0

pip install \
  transformers==5.18.0 accelerate==1.13.0 \
  deepspeed==0.18.3 peft==0.18.1 trl==0.29.1 datasets==4.8.4 \
  flash-linear-attention==0.5.2 ninja

MAX_JOBS=32 NVCC_THREADS=4 pip install --no-build-isolation flash-attn==2.8.3

git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
git fetch origin pull/10338/head
git checkout FETCH_HEAD
pip install -e .
pip install transformers==5.18.0

Reference dependency versions:

Plain Text

python 3.10.0
accelerate 1.13.0
deepspeed 0.18.3
flash_attn 2.8.3
flash-linear-attention 0.5.2
ms_swift 4.6.0.dev0 (PR #10338)
torch 2.8.0+cu128
torchvision 0.23.0
transformers 5.18.0

1B: Data Preparation

Download the dataset from allenai/pixmo-count and convert it into the ms-swift format:

JSON

{
  "messages": [
    {
      "content": "<image>\nCarefully observe the image. Are there any people in the image? If yes, please list their respective coordinates and provide the total count. If no, answer 0.",
      "role": "user"
    },
    {
      "content": "<think>\n\n</think>\n\nThe respective coordinates of people: <point>236 469</point>, <point>307 232</point>, <point>362 434</point>, <point>485 521</point>, <point>487 340</point>, <point>615 386</point>, <point>735 441</point>, <point>870 615</point>. So the total count is 8.",
      "role": "assistant"
    }
  ],
  "images": ["/path/to/images/*.jpg"],
  "source_file": "pixmo-count",
  "orig_index": 1,
  "channel": "pixmo-count"
}

• For the Counting task, adding supervision on point prediction can improve fine-tuning performance. Therefore, we recommend concatenating the points coordinates from the dataset into the assistant response.
• Since MiniCPM-V 4.7 normalizes image coordinates to 0~1000, the point coordinates also need to be transformed as follows:

Python

def expected_norm(x_px: float, y_px: float, width: int, height: int) -> Tuple[int, int]:
    return int((x_px / width) * 1000.0), int((y_px / height) * 1000.0)

1B: Launch Training

After configuring the model path, training set path, validation set path, and output directory, run the following script to start training.

Bash

# run_swift.sh
#!/bin/bash
set -euo pipefail

export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
export NPROC_PER_NODE="${NPROC_PER_NODE:-8}"
export MASTER_PORT="${MASTER_PORT:-29632}"
export WANDB_API_KEY="${WANDB_API_KEY:-}"
export WANDB_PROJECT="${WANDB_PROJECT:-MiniCPMV47-Counting}"
export WANDB_RUN_NAME="${WANDB_RUN_NAME:-minicpmv47_1b_count}"
export WANDB_NAME="${WANDB_NAME:-minicpmv47_1b_count}"
export DOWNSAMPLE_MODE="${DOWNSAMPLE_MODE:-4x}"

GRADIENT_CHECKPOINTING=true
MODE_ARGS=()
if [[ "${DOWNSAMPLE_MODE}" == "16x" ]]; then
    GRADIENT_CHECKPOINTING=false
    MODE_ARGS+=(--ddp_find_unused_parameters true --check_model false)
fi

SWIFT_BIN="${SWIFT_BIN:-swift}"
MODEL_PATH="${MODEL_PATH:-/path/to/MiniCPM-V-4.7-1B}"
TRAIN_DATA="${TRAIN_DATA:-/path/to/task_dataset/train/pixmo_count_train_with_channel}"
VALID_DATA="${VALID_DATA:-/path/to/task_dataset/val/validation-00000-of-00001-swift.parquet}"
DEEPSPEED_CONFIG="${DEEPSPEED_CONFIG:-zero2}"
OUTPUT_DIR="${OUTPUT_DIR:-/path/to/outdir}"

${SWIFT_BIN} sft \
  --model "${MODEL_PATH}" \
  --model_type minicpmv4_7 \
  --template minicpmv4_7 \
  --add_non_thinking_prefix true \
  --run_name "${WANDB_RUN_NAME}" \
  --dataset "${TRAIN_DATA}" \
  --val_dataset "${VALID_DATA}" \
  --deepspeed "${DEEPSPEED_CONFIG}" \
  --tuner_type full \
  --torch_dtype bfloat16 \
  --freeze_llm False \
  --freeze_vit False \
  --freeze_aligner True \
  --packing false \
  --padding_free false \
  --max_length 4096 \
  --num_train_epochs 4 \
  --per_device_train_batch_size 1 \
  --per_device_eval_batch_size 1 \
  --gradient_accumulation_steps 16 \
  --gradient_checkpointing "${GRADIENT_CHECKPOINTING}" \
  --vit_gradient_checkpointing true \
  "${MODE_ARGS[@]}" \
  --learning_rate 5e-6 \
  --warmup_ratio 0.05 \
  --logging_steps 1 \
  --save_steps 132 \
  --eval_strategy steps \
  --eval_steps 80 \
  --save_total_limit 30 \
  --load_from_cache_file false \
  --dataset_num_proc 16 \
  --dataloader_num_workers 16 \
  --enable_channel_loss True \
  --attn_impl flash_attn \
  --loss_scale ignore_empty_think \
  --output_dir "${OUTPUT_DIR}" \
  --report_to none

Key parameter notes:

• Training supports two visual token compression ratios, 16x and 4x, which are controlled by the DOWNSAMPLE_MODE environment variable.
• With the current MiniCPM-V 4.7 template, please keep using --packing false and --padding_free false.
• --add_non_thinking_prefix true can append <think>\n\n</think>\n\n to training prompts, enabling no-thinking training.

1B: Evaluation Results

Metric	Description
Acc@0	Exact match accuracy, i.e. the predicted value is identical to the ground truth
Acc@0 Top1	The highest Acc@0 score among all checkpoints saved during training
Acc@0 Avg.Top3	The average Acc@0 score of the top three checkpoints saved during training

Model	Visual Token Compression Ratio	Acc@0 Top1	Acc@0 Avg.Top3
MiniCPM-V 4.7 1B	16	55.2	N/A
MiniCPM-V 4.7 1B	4	57.7	N/A
Fine-tuned model	16	79.3	79.1
Fine-tuned model	4	82.5	82.5

The original model has not been fine-tuned, so only one Acc@0 result (Acc@0 Top1) is available and Acc@0 Avg.Top3 cannot be computed.

Output example:

Plain Text

Q: Carefully observe the image. Are there any eggs in the image? If yes, please list their
   respective coordinates and provide the total count. If no, answer 0.
A: The respective coordinates of eggs: 252 347, 252 703, 509 351, 517 699, 750 351, 750 670.
   So the total count is 6.

35B-A3B: Environment Setup

Bash

conda create -n "MiniCPM-V-4.7-35B-A3B-Counting" python=3.12 -y
conda activate "MiniCPM-V-4.7-35B-A3B-Counting"

pip install torch==2.10.0 torchvision==0.25.0

pip install \
  transformers==5.18.0 accelerate==1.15.0 \
  deepspeed==0.18.3 peft==0.18.1 trl==0.24.0 \
  flash-linear-attention==0.5.2 wandb ninja einops safetensors tokenizers sentencepiece

MAX_JOBS=32 NVCC_THREADS=4 pip install --no-build-isolation --no-binary causal-conv1d causal-conv1d==1.7.0

git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
git fetch origin pull/10338/head
git checkout FETCH_HEAD
pip install -e .
pip install transformers==5.18.0

Reference dependency versions:

Plain Text

python 3.12.3
accelerate 1.15.0
deepspeed 0.18.3
flash-linear-attention 0.5.2
ms_swift 4.6.0.dev0 (PR #10338)
torch 2.10.0+cu128
torchvision 0.25.0
transformers 5.18.0

35B-A3B: Data Preparation

JSON

{
  "messages": [
    {
      "content": "<image>\nCarefully observe the image. How many people are there in the image? Answer with a single integer only.",
      "role": "user"
    },
    {
      "content": "8",
      "role": "assistant"
    }
  ],
  "images": ["/path/to/images/*.jpg"],
  "source_file": "pixmo-count-train",
  "orig_index": 1,
  "channel": "pixmo-count-no-points"
}

35B-A3B: Launch Training

Bash

# run_swift.sh
#!/bin/bash
set -euo pipefail

export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
export NPROC_PER_NODE="${NPROC_PER_NODE:-8}"
export MASTER_PORT="${MASTER_PORT:-29632}"
export PYTORCH_ALLOC_CONF="${PYTORCH_ALLOC_CONF:-expandable_segments:True}"
export WANDB_API_KEY="${WANDB_API_KEY:-}"
export WANDB_PROJECT="${WANDB_PROJECT:-MiniCPMV47-35B-A3B-Counting}"
export WANDB_RUN_NAME="${WANDB_RUN_NAME:-mcpmv47_35b_count}"
export WANDB_NAME="${WANDB_NAME:-mcpmv47_35b_count}"
export DOWNSAMPLE_MODE="${DOWNSAMPLE_MODE:-4x}"

SWIFT_BIN="${SWIFT_BIN:-swift}"
MODEL_PATH="${MODEL_PATH:-/path/to/MiniCPM-V-4.7-35B-A3B}"
TRAIN_DATA="${TRAIN_DATA:-/path/to/task_dataset/train.jsonl}"
VALID_DATA="${VALID_DATA:-/path/to/task_dataset/validation.jsonl}"
DEEPSPEED_CONFIG="${DEEPSPEED_CONFIG:-zero3}"

if [[ "${DEEPSPEED_CONFIG}" == zero3 ]]; then
    DEEPSPEED_CONFIG="$(python - <<'PY'
import json
from importlib.resources import files
config = json.loads(files('swift').joinpath('config/zero3.json').read_text())
config['zero_optimization']['sub_group_size'] = 100000000
print(json.dumps(config))
PY
)"
fi

OUTPUT_DIR="${OUTPUT_DIR:-path/to/outdir}"

${SWIFT_BIN} sft \
  --model "${MODEL_PATH}" \
  --model_type minicpmv4_7 \
  --template minicpmv4_7 \
  --add_non_thinking_prefix true \
  --run_name "${WANDB_RUN_NAME}" \
  --dataset "${TRAIN_DATA}" \
  --val_dataset "${VALID_DATA}" \
  --deepspeed "${DEEPSPEED_CONFIG}" \
  --experts_impl grouped_mm \
  --tuner_type full \
  --torch_dtype bfloat16 \
  --freeze_vit False \
  --freeze_aligner False \
  --packing false \
  --padding_side left \
  --gradient_checkpointing true \
  --max_length 4096 \
  --num_train_epochs 4 \
  --per_device_train_batch_size "${PER_DEVICE_TRAIN_BATCH_SIZE:-1}" \
  --gradient_accumulation_steps "${GRADIENT_ACCUMULATION_STEPS:-16}" \
  --learning_rate 5e-6 \
  --warmup_ratio 0.05 \
  --logging_steps 1 \
  --save_steps 132 \
  --eval_strategy steps \
  --eval_steps 80 \
  --save_total_limit 30 \
  --load_from_cache_file false \
  --dataset_num_proc 8 \
  --dataloader_num_workers 4 \
  --enable_channel_loss True \
  --attn_impl sdpa \
  --loss_scale ignore_empty_think \
  --output_dir "${OUTPUT_DIR}" \
  --report_to wandb

Key parameter notes:

• Training supports two visual token compression ratios, 16x and 4x, which are controlled by the DOWNSAMPLE_MODE environment variable.
• For this Qwen3.5-based model, please use --packing false to stay consistent with the verified training configuration.
• --add_non_thinking_prefix true can append <think>\n\n</think>\n\n to training prompts, enabling no-thinking training.

35B-A3B: Evaluation Results

Model	Visual Token Compression Ratio	Acc@0 Top1	Acc@0 Avg.Top3
MiniCPM-V 4.7 35B-A3B	16	69.1	N/A
MiniCPM-V 4.7 35B-A3B	4	73.2	N/A
Fine-tuned model	16	81.4	81.3
Fine-tuned model	4	85.0	84.3

Here Acc@0 Top1 is the score on the test set of the checkpoint with the highest Acc@0 on the validation set, and Acc@0 Avg.Top3 is the average score on the test set of the top three checkpoints ranked by the validation set.

Output example:

Plain Text

Q: Carefully observe the image. How many airplanes are there in the image? Answer with a single integer only.
A: 7

LLaMA-Factory

Installation

Bash

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics,deepspeed,minicpm_v]"

Data Construction

Refer to mllm_demo.json under LLaMA-Factory's data/ and construct the image dataset in the same format. If you need to use images in multi-turn conversations, add an <image> tag to the user content of each turn, and the number of <image> tags must match the number of values in images.

JSON

[
  {
    "messages": [
      {"content": "<image>Who are they?", "role": "user"},
      {"content": "They're Kane and Gretzka from Bayern Munich.", "role": "assistant"},
      {"content": "What are they doing?<image>", "role": "user"},
      {"content": "They are celebrating on the soccer field.", "role": "assistant"}
    ],
    "images": ["mllm_demo_data/1.jpg", "mllm_demo_data/1.jpg"]
  }
]

For video datasets, refer to mllm_video_demo.json and use the <video> tag, whose count must match videos:

JSON

[
  {
    "messages": [
      {"content": "<video>Why is this video funny?", "role": "user"},
      {"content": "Because a baby is reading, and he is so cute!", "role": "assistant"}
    ],
    "videos": ["mllm_demo_data/1.mp4"]
  }
]

Register the Dataset

1. Name the JSON file you constructed image_caption.json and place it under the LLaMA-Factory data/ path.
2. Open data/dataset_info.json and search for mllm_demo.
3. Change the key mllm_demo to your own dataset name, such as cpmv_img; change the value of file_name to the name of the dataset file you constructed.

JSON

"cpmv_img": {
  "file_name": "image_caption.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "messages",
    "images": "images"
  },
  "tags": {
    "role_tag": "role",
    "content_tag": "content",
    "user_tag": "user",
    "assistant_tag": "assistant"
  }
}

LoRA Fine-tuning

Create the config file minicpmv4_7_lora_sft.yaml:

YAML

### model
model_name_or_path: openbmb/MiniCPM-V-4.7-1B   # a local model path also works
trust_remote_code: true

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: q_proj,v_proj

### dataset
dataset: cpmv_img          # change to the key you added in data/dataset_info.json
cutoff_len: 3072
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/minicpmv4_7/lora/sft
logging_steps: 1
save_steps: 100
plot_loss: true
overwrite_output_dir: true
save_total_limit: 10

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 1
learning_rate: 1.0e-5
num_train_epochs: 20.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
save_only_model: true

### eval
do_eval: false

Launch training:

Bash

llamafactory-cli train configs/minicpmv4_7_lora_sft.yaml

After training completes, create the merge config minicpmv4_7_lora_export.yaml:

YAML

### model
model_name_or_path: openbmb/MiniCPM-V-4.7-1B      # path to the original model
adapter_name_or_path: saves/minicpmv4_7/lora/sft  # path to the saved LoRA model
finetuning_type: lora
trust_remote_code: true

### export
export_dir: models/minicpmv4_7_lora_sft
export_size: 2
export_device: cpu
export_legacy_format: false

Merge the model:

Bash

llamafactory-cli export configs/minicpmv4_7_lora_export.yaml

Full Fine-tuning

Create the config file minicpmv4_7_full_sft.yaml:

YAML

### model
model_name_or_path: openbmb/MiniCPM-V-4.7-1B
trust_remote_code: true
freeze_vision_tower: true      # freeze the vision module
print_param_status: true
flash_attn: fa2

### method
stage: sft
do_train: true
finetuning_type: full
deepspeed: configs/deepspeed/ds_z2_config.json

### dataset
dataset: cpmv_img
cutoff_len: 3072
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/minicpmv4_7/full/sft
logging_steps: 1
save_steps: 100
plot_loss: true
overwrite_output_dir: true
save_total_limit: 10

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 1
learning_rate: 1.0e-5
num_train_epochs: 20.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
save_only_model: true

### eval
do_eval: false

Launch training:

Bash

llamafactory-cli train configs/minicpmv4_7_full_sft.yaml

The template field of LLaMA-Factory must match the model you use. If the installed version does not yet include a MiniCPM-V 4.7 template, update to a version that supports MiniCPM-V 4.7, or use the ms-swift flow above instead.

Related Resources

After fine-tuning, you can follow Deployment to serve the new weights, or first reduce their size with Quantization.

Topic	Entry
Model overview	MiniCPM-V 4.7 Overview
Code repository	OpenBMB/MiniCPM-V
Official fine-tuning docs	finetune.md
ms-swift framework	modelscope/ms-swift
LLaMA-Factory framework	hiyouga/LLaMA-Factory