MiniCPM-V 4.7

MiniCPM-V 多模态系列

从一张图片和一个问题开始，跑通首次对话，再尝试多图比较、视频理解、OCR、PDF 解析与视觉定位。选择与你任务相符的示例，逐步搭建自己的多模态应用。

MiniCPM-V 4.7 已在 transformers >= 5.18.0 中注册为独立架构 MiniCPMV4_7ForConditionalGeneration，标准 HuggingFace Processor + model.generate 流程即可使用，无需 trust_remote_code。

MiniCPM-V 4.7 有两个尺寸，改 model_path 即可切换：

model_path	尺寸
openbmb/MiniCPM-V-4.7-1B	dense 1B（端侧）
openbmb/MiniCPM-V-4.7-35B-A3B	MoE 35B-A3B（高吞吐）

安装

Bash

pip install "transformers[torch]>=5.18.0" torchvision torchcodec

CUDA 提示

torchcodec（视频解码）对 CUDA 版本较敏感。若遇到 RuntimeError: Could not load libtorchcodec，可改用 PyAV：

pip install "transformers[torch]>=5.18.0" torchvision av

或在装 torch 时指定匹配的 CUDA（如 CUDA 12.8）：

pip install "transformers>=5.18.0" torchvision torchcodec --index-url https://download.pytorch.org/whl/cu128

初始化模型

下面各示例共用这段初始化代码。

Python

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText

model_path = "openbmb/MiniCPM-V-4.7-1B"   # 或 "openbmb/MiniCPM-V-4.7-35B-A3B"

processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForImageTextToText.from_pretrained(
    model_path,
    dtype="auto",
    device_map="auto",
    attn_implementation="sdpa",   # 或 "flash_attention_2"
).eval()

思考模式

MiniCPM-V 4.7 每个尺寸只有一个 checkpoint：对话模板默认带 <think> 块，除非传入 chat_template_kwargs={"enable_thinking": False}。

• 重 OCR / 推理类任务可保留思考，模型会先审视版面再作答。
• 普通问答建议关闭，以获得更快的直答。

单图问答

Python

image = Image.open("./assets/single.png").convert("RGB")

# 第一轮
question = "图中是什么地貌？"
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": question},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=512)
answer = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

多轮对话

把上一轮回答追加回 messages，即可接着提问：

Python

messages.append({"role": "assistant", "content": [{"type": "text", "text": answer}]})
messages.append({
    "role": "user",
    "content": [{"type": "text", "text": "去这里旅行需要注意什么？"}],
})

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=512)
answer = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

多图问答

在 content 里按顺序放多个 image 条目即可：

Python

image1 = Image.open("assets/multi1.png").convert("RGB")
image2 = Image.open("assets/multi2.png").convert("RGB")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image1},
        {"type": "image", "image": image2},
        {"type": "text",  "text":  "比较这两张图片，说明它们之间的差异。"},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=512)
answer = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

视频理解

视频走标准 message 流程：在 content 里放一个 {"type": "video", ...} 条目，由 processor 自动解码 / 采样帧。帧预算与采样由 processor 参数控制 —— 无需手动解帧，也不需要特殊的视频后端。

视频场景建议用 Flash Attention 2，加速并省显存：初始化时传 attn_implementation="flash_attention_2" 与 torch_dtype=torch.bfloat16。

Python

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/football.mp4"},
        {"type": "text", "text": "详细描述这段视频。按时间线展开，重点关注画面文字、界面变化、主要动作与场景切换。"},
    ],
}]

downsample_mode = "16x"   # 需要更精细的细节时用 "4x"

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_num_frames=128,
    stack_frames=1,
    max_slice_nums=1,
    use_image_id=False,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=2048)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

使用本地文件时直接传路径：

JSON

{"type": "video", "url": "./assets/badminton.mp4"}

高级参数

传给 apply_chat_template（标注了 generate 的也需传给 generate）：

参数	默认	适用	说明
downsample_mode	"16x"	图像 & 视频	视觉 token 下采样。"16x" 合并 token 更高效；"4x" 保留 4 倍 token，细节更精细。必须同时传给 generate()。
max_slice_nums	9	图像 & 视频	高分辨率图像切片的最大数量。越大保留细节越多。建议：图像 36，视频 1。
max_num_frames	128	仅视频	控制时间上下文长度、避免显存溢出。短视频（时长 ≤ max_num_frames 秒）按 1 FPS 采样；长视频（> max_num_frames 秒）在全时间线上均匀采样恰好 max_num_frames 帧。
stack_frames	1	仅视频	每秒采样点数量。1 = 仅主帧；N（N>1）= 每秒 1 主帧 + N−1 子帧，子帧合成为一张网格图。短视频建议 1，长视频建议 3~5。
use_image_id	True	图像 & 视频	是否在每个图像 / 帧占位符前加 <image_id>N</image_id>。图像设 True，视频设 False。

downsample_mode 必须同时传给 apply_chat_template（决定占位符数量）和 generate（作用于视觉编码器）。其余参数只传给 apply_chat_template 即可。

OCR 文字识别

Python

ocr_image = Image.open("./assets/ocr.png").convert("RGB")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": ocr_image},
        {"type": "text",  "text":  "图片中的文字是什么？"},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=1024)
answer = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

对中文密集或版面复杂的文本页，开启思考（chat_template_kwargs={"enable_thinking": True}）往往能得到更忠实的转写，因为模型会先审视版面再作答；短文本 / 裁剪良好的图片可关闭思考以更快直答。

PDF 解析

PDF 转图片

Python

# 依赖 poppler-utils：
# sudo apt-get update && sudo apt-get install poppler-utils
from pdf2image import convert_from_path

def pdf_to_images(pdf_path: str, dpi: int = 200):
    images = convert_from_path(pdf_path, dpi=dpi)
    return [image.convert("RGB") for image in images]

逐页解析

Python

prompt = """
You are an OCR assistant. Your task is to identify and extract all visible text from the image provided.
Preserve the original formatting as closely as possible, including:
- Line breaks and paragraphs
- Headings and subheadings
- Any tables, lists, bullet points, or numbered items
- Special characters, spacing, and alignment

Output strictly the extracted text in Markdown format, reflecting the layout and structure of the
original image. Do not add commentary, interpretation, or summarization—only return the raw text
content with its formatting.
"""

images = pdf_to_images("assets/parse.pdf")
content = [{"type": "image", "image": img} for img in images]
content.insert(0, {"type": "text", "text": prompt})
messages = [{"role": "user", "content": content}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=4096)
answer = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

长 PDF 建议逐页（或每次几页）处理以避免显存不足；版面复杂的页面可考虑开启思考。

视觉定位

MiniCPM-V 4.7 返回的定位区域为 \boxed{[x1, y1, x2, y2]}，坐标按 0–1000 归一化。更早的构建使用 <box>x1 y1 x2 y2</box>；下面的解析同时兼容两种格式。

Python

import re
import torch
from PIL import Image, ImageDraw, ImageFont
from transformers import AutoProcessor, AutoModelForImageTextToText

def load_font(size: int):
    # 用真实 TTF 保证标签清晰；找不到时回退到 PIL 内置字体
    for path in (
        "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf",
        "/System/Library/Fonts/Supplemental/Arial Bold.ttf",
    ):
        try:
            return ImageFont.truetype(path, size)
        except OSError:
            continue
    return ImageFont.load_default(size=size)

def extract_bbox(response: str):
    # v4.7：`\boxed{[x1, y1, x2, y2]}`；旧格式：`<box>x1 y1 x2 y2</box>`
    for pat in (r"\\boxed\{\s*\[?\s*([\d,\s]+?)\s*\]?\s*\}", r"<box>\s*([\d\s]+?)\s*</box>"):
        m = re.search(pat, response)
        if m:
            box = [int(v) for v in re.split(r"[,\s]+", m.group(1).strip()) if v]
            if len(box) == 4:
                return box
    raise ValueError(f"Can't find bbox in response: {response!r}")

def draw_bbox(image: Image.Image, bbox, label: str | None = None, color: str = "red"):
    w, h = image.size
    x1, y1, x2, y2 = (
        int(bbox[0] / 1000 * w),
        int(bbox[1] / 1000 * h),
        int(bbox[2] / 1000 * w),
        int(bbox[3] / 1000 * h),
    )
    draw = ImageDraw.Draw(image)
    draw.rectangle([x1, y1, x2, y2], outline=color, width=max(3, round(min(w, h) / 150)))
    if label:
        font = load_font(max(18, round(min(w, h) / 22)))   # 字号随图片缩放
        tb = draw.textbbox((0, 0), label, font=font)
        tw, th = tb[2] - tb[0], tb[3] - tb[1]
        ty = max(0, y1 - th - 8)
        draw.rectangle([x1, ty, x1 + tw + 12, ty + th + 8], fill=color)  # 文字底色
        draw.text((x1 + 6, ty + 2), label, fill="white", font=font)
    return image

model_path = "openbmb/MiniCPM-V-4.7-1B"   # 或 "openbmb/MiniCPM-V-4.7-35B-A3B"
img_path = "./assets/car.png"
question = (
    "Please provide the bounding box coordinate of the region this sentence "
    "describes: <ref>car</ref>"
)

processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForImageTextToText.from_pretrained(
    model_path, dtype="auto", device_map="auto", attn_implementation="sdpa",
).eval()

image = Image.open(img_path).convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": question},
    ],
}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

with torch.inference_mode():
    out_ids = model.generate(**inputs, max_new_tokens=512)

response = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
)
print("Model output:", response)

bbox = extract_bbox(response)
draw_bbox(image, bbox, label="car").save("./assets/car_grounding.png")

相关资源

主题	入口
代码仓库	OpenBMB/MiniCPM-V
在线体验	MiniCPM-V 4.7 Demo
API 接入	API 使用文档