MiniCPM-V 4.5

MiniCPM-V 多模态系列

用更小的权重换取更低的显存与存储占用。GGUF 适用于本地与 CPU / Metal 运行，BNB / AWQ / GPTQ 面向 GPU 推理。可直接使用官方预量化权重，也可按本页脚本自行量化。

格式	精度	工具	适用场景
GGUF	Q4_K_M 等多档	llama.cpp	llama.cpp / Ollama 本地运行
BNB	int4	bitsandbytes	GPU 推理，脚本最简单
AWQ	4-bit	AutoAWQ	GPU 推理，需校准数据
GPTQ	4-bit	AutoGPTQ	GPU 推理，需校准数据

GGUF

本指南将引导您完成将 MiniCPM-V 4.5 PyTorch 模型转换为 GGUF 格式并进行量化的过程。

本指南涉及的模型需要基于 llama.cpp/ollama 使用。

1. 下载 PyTorch 模型

首先，请从以下任一来源获取原始的 PyTorch 模型文件：

• HuggingFace: https://huggingface.co/openbmb/MiniCPM-V-4_5
• 魔搭社区: https://modelscope.cn/models/OpenBMB/MiniCPM-V-4_5

2. 将 PyTorch 模型转换为 GGUF 格式

请依次执行以下命令，以完成模型结构调整、视觉编码器转换和语言模型转换。

Bash

# 步骤 1: 对模型结构进行预处理
python ./tools/mtmd/legacy-models/minicpmv-surgery.py -m ../MiniCPM-V-4_5

# 步骤 2: 将视觉编码器转换为 GGUF 格式
python ./tools/mtmd/legacy-models/minicpmv-convert-image-encoder-to-gguf.py -m ../MiniCPM-V-4_5 --minicpmv-projector ../MiniCPM-V-4_5/minicpmv.projector --output-dir ../MiniCPM-V-4_5/ --minicpmv_version 6

# 步骤 3: 将语言模型转换为 GGUF 格式
python ./convert-hf-to-gguf.py ../MiniCPM-V-4_5/model

3. 执行 INT4 量化

转换完成后，使用 llama-quantize 工具对 F16 精度的 GGUF 模型进行 INT4 量化。

Bash

./llama-quantize ../MiniCPM-V-4_5/model/ggml-model-f16.gguf ../MiniCPM-V-4_5/model/ggml-model-Q4_K_M.gguf Q4_K_M

BNB

支持版本： MiniCPM-V 4.5 / MiniCPM-V 4.0 / MiniCPM-V 2.6 / MiniCPM-V 2.5

1. 下载模型

从 HuggingFace 下载 MiniCPM-V-4_5 模型，解压到本地目录。

2. 量化脚本

下方脚本会加载原始模型，使用 bitsandbytes 量化为 4-bit，并保存量化后的模型。

Python

import torch
from transformers import AutoModel, AutoTokenizer, BitsAndBytesConfig
from PIL import Image
import time
import GPUtil
import os

assert torch.cuda.is_available(), "CUDA is not available, but this code requires a GPU."

device = 'cuda'  # 选择 GPU
model_path = '/model/MiniCPM-V-4_5' # 模型路径
save_path = './model/MiniCPM-V-4_5-int4' # 量化模型保存路径
image_path = './assets/airplane.jpeg'

# 量化配置
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    load_in_8bit=False,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_storage=torch.uint8,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    llm_int8_enable_fp32_cpu_offload=False,
    llm_int8_has_fp16_weight=False,
    llm_int8_skip_modules=["out_proj", "kv_proj", "lm_head"],
    llm_int8_threshold=6.0
)

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_path,
    device_map=device,
    quantization_config=quantization_config,
    trust_remote_code=True
)

gpu_usage = GPUtil.getGPUs()[0].memoryUsed
start = time.time()
response = model.chat(
    image=Image.open(image_path).convert("RGB"),
    msgs=[
        {
            "role": "user",
            "content": "图中是什么？"
        }
    ],
    tokenizer=tokenizer
)
print('量化后输出：', response)
print('量化后推理耗时：', time.time() - start)
print(f"量化后 GPU 显存占用：{round(gpu_usage/1024, 2)}GB")

# 保存量化模型与 tokenizer
os.makedirs(save_path, exist_ok=True)
model.save_pretrained(save_path, safe_serialization=True)
tokenizer.save_pretrained(save_path)

3. 预期输出

量化完成后大致会看到：

Plain Text

量化后输出： The image depicts an Airbus A380-800 aircraft in mid-flight against a clear blue sky. The airplane is predominantly white with a distinctive blue tail fin featuring a red and white logo. The fuselage has the text "Airbus A380-800" written on it, and there are Chinese characters along the side of the aircraft. The landing gear is partially extended, indicating that the plane is either taking off or preparing to land. The engines are visible under the wings, and the overall design showcases the large size and advanced engineering of the Airbus A380 model.
量化后推理耗时： 9.316158771514893
量化后 GPU 显存占用：18.97GB

量化模型会保存到 save_path 指定的目录，可用于继续微调或推理。

---

要自定义模型路径、图片或保存路径，修改脚本中对应的变量即可。

AWQ

支持版本： MiniCPM-V 4.5

方法 1（用预量化模型 + vLLM 推理）

1. 下载模型

从 HuggingFace 下载 AutoAWQ 量化的 4-bit MiniCPM-V-4_5 模型。

Bash

git clone https://huggingface.co/openbmb/MiniCPM-V-4_5-AWQ

2. 用 vLLM 运行

Python

import os
from PIL import Image
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams


# 量化模型名称或本地路径
MODEL_NAME = "openbmb/MiniCPM-V-4_5-AWQ"

# 图片文件列表
IMAGES = [
    "image.png",
]

# 加载并转换图片
image = Image.open(IMAGES[0]).convert("RGB")

# 初始化 tokenizer
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)

# 初始化 LLM
llm = LLM(
    model=MODEL_NAME,
    # gpu_memory_utilization=0.9,
    max_model_len=2048,
    trust_remote_code=True,
    # disable_mm_preprocessor_cache=True,
    # limit_mm_per_prompt={"image": 5}
)

# 构建消息
messages = [{
    "role": "user",
    "content": "(<image>./</image>)\n请描述这张图片的内容",
    # "content": "(<image>./</image>)\nPlease describe the content of this image",
}]

# 应用 chat template
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# 设置 stop token IDs
stop_tokens = ['<|im_end|>', '</s>']
stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]

# 采样参数
sampling_params = SamplingParams(
    stop_token_ids=stop_token_ids,
    temperature=0.7,
    # detokenize=True,
    top_p=0.8,
    # top_k=100,
    # seed=3472,
    max_tokens=1024,
    # min_tokens=150,
)

# 推理
outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": {
        "image": image
    }
}, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

方法 2（直接用 AWQ 量化模型推理）

1. 下载模型

从 HuggingFace 下载 MiniCPM-V 4.5 模型。

Bash

git clone https://huggingface.co/openbmb/MiniCPM-V-4_5

2. 从源码安装 AutoAWQ

官方 AutoAWQ 仓库已不再维护，请下载并从源码安装我们的 fork。

Bash

git clone https://github.com/tc-mb/AutoAWQ.git
cd AutoAWQ
pip install -e .

3. 推理脚本

直接使用 AWQ 量化模型推理：

Python

import os
from PIL import Image
from transformers import AutoTokenizer, TextStreamer
from awq import AutoAWQForCausalLM
import torch

# 量化模型名称或本地路径
model_path = "openbmb/MiniCPM-V-4_5-AWQ"
device = 'cuda'
# 图片文件路径
image_path = './assets/airplane.jpeg'

model = AutoAWQForCausalLM.from_quantized(model_path, trust_remote_code=True).to('cuda')
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

gpu_usage = GPUtil.getGPUs()[0].memoryUsed
response = model.chat(
    image=Image.open(image_path).convert("RGB"),
    msgs=[
        {
            "role": "user",
            "content": "图中是什么？"
        }
    ],
    tokenizer=tokenizer
) # 模型推理

print('输出:', response)

方法 3（自己进行 AWQ 量化）

1. 下载模型

从 HuggingFace 下载 MiniCPM-V 4.5 模型。

Bash

git clone https://huggingface.co/openbmb/MiniCPM-V-4_5

2. 从源码安装 AutoAWQ

官方 AutoAWQ 仓库已不再维护，请下载并从源码安装我们的 fork。

Bash

git clone https://github.com/tc-mb/AutoAWQ.git
cd AutoAWQ
pip install -e .

3. 量化脚本

执行下方量化脚本（按需替换 model_path 和 quant_path 为原始模型与量化模型保存路径）：

Python

import os
from datasets import load_dataset, load_from_disk
from awq import AutoAWQForCausalLM
import torch
from transformers import AutoTokenizer
import shutil

# 原始模型路径（可以是本地路径或 HF 模型 ID）
model_path = '/openbmb/MiniCPM-V-4_5'

# 量化模型保存路径
quant_path = '/model_quantized/minicpmv4_5_awq'

# 量化配置：4-bit 权重，group size 128，GEMM 后端
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" } # "w_bit":4 or 8


# 加载原始模型与 tokenizer
model = AutoAWQForCausalLM.from_pretrained(model_path, trust_remote_code=True, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# 把 model_path 中存在但 quant_path 中没有的文件复制过去（跳过权重文件）
def copy_files_not_in_B(A_path, B_path):
    """
    如果文件在 A 中存在但 B 中不存在，则从 A 拷贝到 B。

    :param A_path: 源目录 (A)
    :param B_path: 目标目录 (B)
    """
    # 确保源目录存在
    if not os.path.exists(A_path):
        raise FileNotFoundError(f"The directory {A_path} does not exist.")
    if not os.path.exists(B_path):
        os.makedirs(B_path)

    # 列出 A 中所有文件，排除权重文件 (.bin / safetensors)
    files_in_A = os.listdir(A_path)
    files_in_A = set([file for file in files_in_A if not (".bin" in file or "safetensors" in file )])
    # 列出 B 中所有文件
    files_in_B = set(os.listdir(B_path))

    # 计算需要复制的文件
    files_to_copy = files_in_A - files_in_B

    # 逐个复制
    for file in files_to_copy:
        src_file = os.path.join(A_path, file)
        dst_file = os.path.join(B_path, file)
        if os.path.isfile(src_file):
            shutil.copy2(src_file, dst_file)

# 数据加载方法
# 加载 Alpaca 数据集
def load_alpaca():
    data = load_dataset("tatsu-lab/alpaca", split="train")

    # 把每条样本转换成 chat 风格的 prompt
    def concatenate_data(x):
        if x['input'] and x['instruction']:
            msgs = [
                    {"role": "system", "content": x['instruction']},
                    {"role": "user", "content": x['input']},
                    {"role": "assistant", "content": x['output']},
            ]
        elif x['input']:
            msgs = [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": x['input']},
                {"role": "assistant", "content": x['output']}
            ]
        else:
            msgs = [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": x['instruction']},
                {"role": "assistant", "content": x['output']}
            ]

        data = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
        return {"text": data}

    concatenated = data.map(concatenate_data)
    return [text for text in concatenated["text"]][:1024]

# 加载 Wikitext 数据集
def load_wikitext():
    data = load_dataset('wikitext', 'wikitext-2-raw-v1', split="train")
    return [text for text in data["text"] if text.strip() != '' and len(text.split(' ')) > 20]


# 加载校准数据
calib_data = load_alpaca()
# 量化
model.quantize(tokenizer, quant_config=quant_config, calib_data=calib_data)

# shutil.rmtree(quant_path, ignore_errors=True)

# 保存量化模型
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

copy_files_not_in_B(model_path, quant_path)
print(f'Model is quantized and saved at "{quant_path}"')

GPTQ

支持版本： MiniCPM-V 4.5

方法 1（用预量化模型）

1. 下载模型

从下面任一来源获取 4-bit GPTQ 量化的 MiniCPM-V-4_5：

• HuggingFace — <https://huggingface.co/openbmb/MiniCPM-V-4_5-GPTQ>
• ModelScope — <https://modelscope.cn/models/OpenBMB/MiniCPM-V-4_5-GPTQ>

Bash

git clone https://huggingface.co/openbmb/MiniCPM-V-4_5-GPTQ

2. 用 vLLM 运行

Python

import os
from PIL import Image
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams


# 量化模型名称或本地路径
MODEL_NAME = "openbmb/MiniCPM-V-4_5-GPTQ"

# 图片文件列表
IMAGES = [
    "image.png",
]

# 加载并转换图片
image = Image.open(IMAGES[0]).convert("RGB")

# 初始化 tokenizer
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)

# 初始化 LLM
llm = LLM(
    model=MODEL_NAME,
    # gpu_memory_utilization=0.9,
    max_model_len=2048,
    trust_remote_code=True,
    # disable_mm_preprocessor_cache=True,
    # limit_mm_per_prompt={"image": 5}
)

# 构建消息
messages = [{
    "role": "user",
    "content": "(<image>./</image>)\n请描述这张图片的内容",
    # "content": "(<image>./</image>)\nPlease describe the content of this image",
}]

# 应用 chat template
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# 设置 stop token IDs
stop_tokens = ['<|im_end|>', '</s>']
stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]

# 采样参数
sampling_params = SamplingParams(
    stop_token_ids=stop_token_ids,
    temperature=0.7,
    top_p=0.8,
    max_tokens=1024,
)

# 推理
outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": {
        "image": image
    }
}, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

方法 2（自己进行 GPTQ 量化）

1. 下载模型

从 HuggingFace 下载 MiniCPM-V 4.5 模型。

Bash

git clone https://huggingface.co/openbmb/MiniCPM-V-4_5

2. 安装 AutoGPTQ

Bash

git clone https://github.com/tc-mb/AutoGPTQ.git
cd AutoGPTQ
pip install -e .

3. 量化脚本

下方脚本会从 MiniCPM-V-4_5 中提取 LLM backbone（Qwen3），用 GPTQ 量化为 4-bit，再把量化后的权重与原始多模态部分重新组装。

执行下方脚本（按需替换 MODEL_PATH 与 OUTPUT_PATH）：

Python

import os
import sys
import json
import shutil
import logging
from pathlib import Path

import torch
from datasets import load_dataset
from transformers import AutoModel, AutoTokenizer, AutoConfig


def _ensure_hf_dynamic_module_cache(model_path: str) -> None:
    """提前复制所有 .py 文件，确保间接相对导入能解析。"""
    from transformers.dynamic_module_utils import (
        HF_MODULES_CACHE,
        TRANSFORMERS_DYNAMIC_MODULE_NAME,
        _sanitize_module_name,
        init_hf_modules,
    )
    init_hf_modules()
    submodule = _sanitize_module_name(Path(model_path).name)
    cache_dir = Path(HF_MODULES_CACHE) / TRANSFORMERS_DYNAMIC_MODULE_NAME / submodule
    cache_dir.mkdir(parents=True, exist_ok=True)
    for py in Path(model_path).glob("*.py"):
        dst = cache_dir / py.name
        if not dst.exists():
            shutil.copy2(str(py), str(dst))


logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
    level=logging.INFO,
    datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(__name__)

MODEL_PATH = '/model/MiniCPM-V-4_5'
OUTPUT_PATH = './model/MiniCPM-V-4_5-gptq-int4'

BITS = 4
GROUP_SIZE = 128
NUM_CALIBRATION_SAMPLES = 128
MAX_SEQ_LEN = 512


def prepare_calibration_data(tokenizer, num_samples, max_length):
    """从 Alpaca 数据集准备校准数据。"""
    dataset = load_dataset("tatsu-lab/alpaca", split="train")
    dataset = dataset.shuffle(seed=42).select(range(min(num_samples, len(dataset))))

    examples = []
    for sample in dataset:
        text = sample.get("text", "") or sample.get("output", "") or sample.get("instruction", "")
        if not text:
            continue
        tokenized = tokenizer(
            text,
            truncation=True,
            max_length=max_length,
            return_tensors="pt",
        )
        examples.append({
            "input_ids": tokenized["input_ids"].squeeze(0),
            "attention_mask": tokenized["attention_mask"].squeeze(0),
        })

    logger.info(f"Prepared {len(examples)} calibration samples")
    return examples


def main():
    logger.info(f"Loading tokenizer from {MODEL_PATH}")
    tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)

    logger.info("Preparing calibration data...")
    examples = prepare_calibration_data(tokenizer, NUM_CALIBRATION_SAMPLES, MAX_SEQ_LEN)

    _ensure_hf_dynamic_module_cache(MODEL_PATH)
    logger.info(f"Loading full MiniCPM-V model from {MODEL_PATH}")
    full_model = AutoModel.from_pretrained(
        MODEL_PATH,
        trust_remote_code=True,
        torch_dtype=torch.float16,
        device_map="cpu",
    )

    llm = full_model.llm
    logger.info(f"Extracted LLM backbone: {llm.__class__.__name__}")

    original_model_type = llm.config.model_type
    llm.config.model_type = "qwen3"

    from auto_gptq import BaseQuantizeConfig
    from auto_gptq.modeling.qwen3 import Qwen3GPTQForCausalLM

    quantize_config = BaseQuantizeConfig(
        bits=BITS,
        group_size=GROUP_SIZE,
        desc_act=False,
        sym=True,
    )
    logger.info(f"Quantize config: bits={BITS}, group_size={GROUP_SIZE}")

    llm.seqlen = MAX_SEQ_LEN
    gptq_model = Qwen3GPTQForCausalLM(llm, False, quantize_config)

    logger.info("Starting GPTQ quantization...")
    gptq_model.quantize(examples, batch_size=1)
    logger.info("Quantization complete!")

    os.makedirs(OUTPUT_PATH, exist_ok=True)

    llm_temp_dir = os.path.join(OUTPUT_PATH, "_llm_temp")
    os.makedirs(llm_temp_dir, exist_ok=True)
    gptq_model.save_quantized(llm_temp_dir, use_safetensors=True)

    # 把原始模型中的非权重文件复制过来
    for fname in os.listdir(MODEL_PATH):
        src = os.path.join(MODEL_PATH, fname)
        dst = os.path.join(OUTPUT_PATH, fname)
        if fname.startswith("model") and fname.endswith(".safetensors"):
            continue
        if fname in ("config.json", "quantize_config.json", "model.safetensors.index.json"):
            continue
        if os.path.isfile(src) and not os.path.exists(dst):
            shutil.copy2(src, dst)
        elif os.path.isdir(src) and not os.path.exists(dst):
            shutil.copytree(src, dst)

    # 把量化后的 LLM 权重和原始多模态权重组合
    from safetensors.torch import load_file as safe_load, save_file as safe_save

    quantized_llm_files = [f for f in os.listdir(llm_temp_dir) if f.endswith(".safetensors")]
    llm_state_dict = {}
    for f in quantized_llm_files:
        sd = safe_load(os.path.join(llm_temp_dir, f))
        llm_state_dict.update(sd)

    non_llm_state_dict = {}
    full_model_sd = full_model.state_dict()
    for key, value in full_model_sd.items():
        if not key.startswith("llm."):
            non_llm_state_dict[key] = value

    combined_sd = {}
    for key, value in llm_state_dict.items():
        combined_sd[f"llm.{key}"] = value
    for key, value in non_llm_state_dict.items():
        combined_sd[key] = value

    MAX_SHARD_SIZE_BYTES = 4 * 1024 * 1024 * 1024  # 每分片 4GB
    sorted_keys = sorted(combined_sd.keys())

    shards = []
    current_shard = {}
    current_size = 0
    weight_map = {}

    for key in sorted_keys:
        tensor = combined_sd[key]
        tensor_size = tensor.nelement() * tensor.element_size()
        if current_size > 0 and current_size + tensor_size > MAX_SHARD_SIZE_BYTES:
            shards.append(current_shard)
            current_shard = {}
            current_size = 0
        current_shard[key] = tensor.clone().contiguous()
        current_size += tensor_size

    if current_shard:
        shards.append(current_shard)

    if len(shards) == 1:
        shard_name = "model.safetensors"
        safe_save(shards[0], os.path.join(OUTPUT_PATH, shard_name))
        for key in shards[0]:
            weight_map[key] = shard_name
    else:
        for i, shard in enumerate(shards):
            shard_name = f"model-{i+1:05d}-of-{len(shards):05d}.safetensors"
            safe_save(shard, os.path.join(OUTPUT_PATH, shard_name))
            for key in shard:
                weight_map[key] = shard_name

    index = {
        "metadata": {"total_size": sum(t.nelement() * t.element_size() for t in combined_sd.values())},
        "weight_map": weight_map,
    }
    with open(os.path.join(OUTPUT_PATH, "model.safetensors.index.json"), "w") as f:
        json.dump(index, f, indent=2)

    # 保存带量化信息的 config
    config = AutoConfig.from_pretrained(MODEL_PATH, trust_remote_code=True)
    config_dict = config.to_dict()
    config_dict["quantization_config"] = {
        "bits": BITS,
        "group_size": GROUP_SIZE,
        "damp_percent": 0.01,
        "desc_act": False,
        "static_groups": False,
        "sym": True,
        "true_sequential": True,
        "quant_method": "gptq",
        "checkpoint_format": "gptq",
        "model_name_or_path": None,
        "model_file_base_name": None,
    }
    config_dict["model_type"] = original_model_type

    with open(os.path.join(OUTPUT_PATH, "config.json"), "w") as f:
        json.dump(config_dict, f, indent=2)

    quant_config_dict = quantize_config.to_dict()
    with open(os.path.join(OUTPUT_PATH, "quantize_config.json"), "w") as f:
        json.dump(quant_config_dict, f, indent=2)

    shutil.rmtree(llm_temp_dir)

    del combined_sd, llm_state_dict, non_llm_state_dict, full_model_sd
    torch.cuda.empty_cache()

    logger.info(f"Done! Quantized model saved to: {OUTPUT_PATH}")
    logger.info(f"Quantization: W{BITS}A16 GPTQ (weight-only {BITS}-bit, activation fp16)")
    logger.info("Compatible with: transformers (via optimum) and vLLM")


if __name__ == "__main__":
    main()