MiniCPM-V 4.7

MiniCPM-V Multimodal Series

Start from a single image and one question to get your first conversation working, then try multi-image comparison, video understanding, OCR, PDF parsing and visual grounding. Pick the example that matches your task and build your multimodal app from there.

MiniCPM-V 4.7 is registered in transformers >= 5.18.0 as the standalone architecture MiniCPMV4_7ForConditionalGeneration, so the standard HuggingFace Processor + model.generate flow works out of the box, with no trust_remote_code needed.

MiniCPM-V 4.7 comes in two sizes — just change model_path to switch:

`model_path`	Size
`openbmb/MiniCPM-V-4.7-1B`	dense 1B (on-device)
`openbmb/MiniCPM-V-4.7-35B-A3B`	MoE 35B-A3B (high throughput)

Install

Bash

pip install "transformers[torch]>=5.18.0" torchvision torchcodec

CUDA note

torchcodec (video decoding) is sensitive to the CUDA version. If you hit RuntimeError: Could not load libtorchcodec, switch to PyAV instead:

pip install "transformers[torch]>=5.18.0" torchvision av

Or pin the matching CUDA build (e.g. CUDA 12.8) when installing torch:

pip install "transformers>=5.18.0" torchvision torchcodec --index-url https://download.pytorch.org/whl/cu128

Initialize model

The examples below all share this initialization code.

Python

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText

model_path = "openbmb/MiniCPM-V-4.7-1B"   # or "openbmb/MiniCPM-V-4.7-35B-A3B"

processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForImageTextToText.from_pretrained(
    model_path,
    dtype="auto",
    device_map="auto",
    attn_implementation="sdpa",   # or "flash_attention_2"
).eval()

Thinking mode

MiniCPM-V 4.7 has only one checkpoint per size: the chat template includes a <think> block by default unless you pass chat_template_kwargs={"enable_thinking": False}.

OCR-heavy / reasoning tasks can keep thinking on — the model inspects the layout before answering.
Ordinary Q&A should turn it off for a faster direct answer.

Single-image QA

Python

image = Image.open("./assets/single.png").convert("RGB")

# First round
question = "What is the landform in the picture?"
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": question},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=512)
answer = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

Multi-turn conversation

Append the previous answer back to messages to keep asking:

Python

messages.append({"role": "assistant", "content": [{"type": "text", "text": answer}]})
messages.append({
    "role": "user",
    "content": [{"type": "text", "text": "What should I pay attention to when traveling here?"}],
})

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=512)
answer = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

Multi-image QA

Just put several image entries in content, in order:

Python

image1 = Image.open("assets/multi1.png").convert("RGB")
image2 = Image.open("assets/multi2.png").convert("RGB")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image1},
        {"type": "image", "image": image2},
        {"type": "text",  "text":  "Compare the two images, tell me about the differences between them."},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=512)
answer = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

Video understanding

Video goes through the standard message flow: put a {"type": "video", ...} entry in content and let the processor decode / sample frames automatically. Frame budget and sampling are controlled by processor parameters — no manual frame decoding and no special video backend required.

For video scenarios, Flash Attention 2 is recommended for acceleration and memory savings: pass attn_implementation="flash_attention_2" and torch_dtype=torch.bfloat16 at initialization.

Python

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/football.mp4"},
        {"type": "text", "text": "Describe this video in detail. Follow the timeline and focus on on-screen text, interface changes, main actions, and scene changes."},
    ],
}]

downsample_mode = "16x"   # use "4x" when you need finer detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_num_frames=128,
    stack_frames=1,
    max_slice_nums=1,
    use_image_id=False,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=2048)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

To use a local file, pass the path directly:

JSON

{"type": "video", "url": "./assets/badminton.mp4"}

Advanced parameters

Pass these to apply_chat_template (those marked generate must also be passed to generate):

Parameter	Default	Applies to	Description
`downsample_mode`	`"16x"`	Image & Video	Visual token downsampling. `"16x"` merges tokens for efficiency; `"4x"` keeps 4× more tokens for finer detail. Must also be passed to `generate()`.
`max_slice_nums`	`9`	Image & Video	Maximum number of slices when splitting a high-resolution image. Higher values preserve more detail. Recommended: `36` for image, `1` for video.
`max_num_frames`	`128`	Video only	Controls the temporal context length and prevents VRAM overflow. Short videos (≤ `max_num_frames` sec): sampled at 1 FPS. Long videos (> `max_num_frames` sec): uniformly sampled to exactly `max_num_frames` frames.
`stack_frames`	`1`	Video only	Sample points per second. `1` = main frame only. `N` (N>1) = 1 main frame + N−1 sub-frames per second, composited into a grid image. Recommended `1` for short videos, `3`–`5` for long videos.
`use_image_id`	`True`	Image & Video	Whether to prepend `<image_id>N</image_id>` before each image/frame placeholder. Set `True` for image, `False` for video.

downsample_mode must be passed to both apply_chat_template (it determines the placeholder count) and generate (it acts on the vision encoder). All other parameters only need to go to apply_chat_template.

OCR

Python

ocr_image = Image.open("./assets/ocr.png").convert("RGB")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": ocr_image},
        {"type": "text",  "text":  "What is the text in the picture?"},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=1024)
answer = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

For Chinese-heavy or dense-layout text pages, enabling thinking (chat_template_kwargs={"enable_thinking": True}) tends to give more faithful transcriptions, because the model inspects the layout before answering; for short text / well-cropped images you can turn thinking off for a faster direct answer.

PDF parsing

PDF to images

Python

# requires poppler-utils:
#   sudo apt-get update && sudo apt-get install poppler-utils
from pdf2image import convert_from_path

def pdf_to_images(pdf_path: str, dpi: int = 200):
    images = convert_from_path(pdf_path, dpi=dpi)
    return [image.convert("RGB") for image in images]

Page-by-page parsing

Python

prompt = """
You are an OCR assistant. Your task is to identify and extract all visible text from the image provided.
Preserve the original formatting as closely as possible, including:
- Line breaks and paragraphs
- Headings and subheadings
- Any tables, lists, bullet points, or numbered items
- Special characters, spacing, and alignment

Output strictly the extracted text in Markdown format, reflecting the layout and structure of the
original image. Do not add commentary, interpretation, or summarization—only return the raw text
content with its formatting.
"""

images = pdf_to_images("assets/parse.pdf")
content = [{"type": "image", "image": img} for img in images]
content.insert(0, {"type": "text", "text": prompt})
messages = [{"role": "user", "content": content}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=4096)
answer = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

For long PDFs, process page by page (or a few pages at a time) to avoid running out of VRAM; for pages with complex layouts, consider enabling thinking.

Visual grounding

The grounding regions returned by MiniCPM-V 4.7 are \boxed{[x1, y1, x2, y2]}, with coordinates normalized to 0–1000. Earlier builds use <box>x1 y1 x2 y2</box>; the parser below handles both formats.

Python

import re
import torch
from PIL import Image, ImageDraw, ImageFont
from transformers import AutoProcessor, AutoModelForImageTextToText

def load_font(size: int):
    # use a real TTF for crisp labels; fall back to PIL's built-in font if none is found
    for path in (
        "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf",
        "/System/Library/Fonts/Supplemental/Arial Bold.ttf",
    ):
        try:
            return ImageFont.truetype(path, size)
        except OSError:
            continue
    return ImageFont.load_default(size=size)

def extract_bbox(response: str):
    # v4.7: `\boxed{[x1, y1, x2, y2]}`; legacy format: `<box>x1 y1 x2 y2</box>`
    for pat in (r"\\boxed\{\s*\[?\s*([\d,\s]+?)\s*\]?\s*\}", r"<box>\s*([\d\s]+?)\s*</box>"):
        m = re.search(pat, response)
        if m:
            box = [int(v) for v in re.split(r"[,\s]+", m.group(1).strip()) if v]
            if len(box) == 4:
                return box
    raise ValueError(f"Can't find bbox in response: {response!r}")

def draw_bbox(image: Image.Image, bbox, label: str | None = None, color: str = "red"):
    w, h = image.size
    x1, y1, x2, y2 = (
        int(bbox[0] / 1000 * w),
        int(bbox[1] / 1000 * h),
        int(bbox[2] / 1000 * w),
        int(bbox[3] / 1000 * h),
    )
    draw = ImageDraw.Draw(image)
    draw.rectangle([x1, y1, x2, y2], outline=color, width=max(3, round(min(w, h) / 150)))
    if label:
        font = load_font(max(18, round(min(w, h) / 22)))   # font size scales with the image
        tb = draw.textbbox((0, 0), label, font=font)
        tw, th = tb[2] - tb[0], tb[3] - tb[1]
        ty = max(0, y1 - th - 8)
        draw.rectangle([x1, ty, x1 + tw + 12, ty + th + 8], fill=color)  # label background
        draw.text((x1 + 6, ty + 2), label, fill="white", font=font)
    return image

model_path = "openbmb/MiniCPM-V-4.7-1B"   # or "openbmb/MiniCPM-V-4.7-35B-A3B"
img_path = "./assets/car.png"
question = (
    "Please provide the bounding box coordinate of the region this sentence "
    "describes: <ref>car</ref>"
)

processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForImageTextToText.from_pretrained(
    model_path, dtype="auto", device_map="auto", attn_implementation="sdpa",
).eval()

image = Image.open(img_path).convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": question},
    ],
}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
    chat_template_kwargs={"enable_thinking": False},
).to(model.device)

with torch.inference_mode():
    out_ids = model.generate(**inputs, max_new_tokens=512)

response = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
)
print("Model output:", response)

bbox = extract_bbox(response)
draw_bbox(image, bbox, label="car").save("./assets/car_grounding.png")

Topic	Link
Code repository	OpenBMB/MiniCPM-V
Online demo	MiniCPM-V 4.7 Demo
API access	API guide

Developer Showcase

View all →

Install​

Initialize model​

Thinking mode​

Single-image QA​

Multi-turn conversation​

Multi-image QA​

Video understanding​

Advanced parameters​

OCR​

PDF parsing​

PDF to images​

Page-by-page parsing​

Visual grounding​

Related Resources​

Developer Showcase

Install

Initialize model

Thinking mode

Single-image QA

Multi-turn conversation

Multi-image QA

Video understanding

Advanced parameters

OCR

PDF parsing

PDF to images

Page-by-page parsing

Visual grounding

Related Resources