MiniCPM-V 4.5

MiniCPM-V 多模态系列

从一张图片和一个问题开始，跑通首次对话，再尝试多图比较、视频理解、OCR、PDF 解析与视觉定位。选择与你任务相符的示例，逐步搭建自己的多模态应用。

MiniCPM-V 4.5 是单一权重，通过 model.chat() 的 enable_thinking 参数在非思考与思考模式间切换：非思考模式直接作答、延迟更低，思考模式先推理再作答、更适合需要多步判断的题目。本页的「单图 / 多图（思考模式）」即为后者的示例。

加载模型统一使用 AutoModel.from_pretrained(..., trust_remote_code=True)，注意力实现请用 sdpa 或 flash_attention_2，不要用 eager。

单图问答

加载模型

Python

from PIL import Image
import torch
from transformers import AutoModel, AutoTokenizer

model_path = 'openbmb/MiniCPM-V-4_5'
model = AutoModel.from_pretrained(model_path, trust_remote_code=True,
                                  # sdpa 或 flash_attention_2，不要用 eager
                                  attn_implementation='sdpa', torch_dtype=torch.bfloat16)
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(
    model_path, trust_remote_code=True)

单图对话

Python

image = Image.open('./assets/single.png').convert('RGB')

# 第一轮对话
question = "图片中的地貌是什么？"
msgs = [{'role': 'user', 'content': [image, question]}]

answer = model.chat(
    msgs=msgs,
    image=image,
    tokenizer=tokenizer
)
print(answer)

第二轮对话

Python

# 第二轮对话，带上多轮历史上下文
msgs.append({"role": "assistant", "content": [answer]})
msgs.append({"role": "user", "content": [
            "去这里旅行需要注意什么？"]})

answer = model.chat(
    msgs=msgs,
    image=None,
    tokenizer=tokenizer
)
print(answer)

示例图片

示例输出

第一轮

Plain Text

The landform in the picture is a mountain range. The image shows steep cliffs, rugged terrain, and a winding path that ascends between two prominent rocky peaks covered with green vegetation. This type of landscape features high elevations and dramatic slopes characteristic of mountains.

第二轮

Plain Text

When traveling to a mountainous area like this, it's important to pay attention to several factors. Firstly, be mindful of the steep paths and uneven terrain; wear appropriate footwear with good grip for safety. Secondly, stay hydrated as high altitudes can lead to dehydration more quickly than at sea level. Thirdly, check weather conditions before your trip since mountains can have sudden changes in weather which might affect visibility and trail conditions. Lastly, if you plan to hike or climb, carry essential supplies such as food, water, first aid kits, and navigation tools (maps or GPS) because help may not be readily available on remote trails.

多图问答

加载模型

Python

from PIL import Image
import torch
from transformers import AutoModel, AutoTokenizer

model_path = 'openbmb/MiniCPM-V-4_5'
model = AutoModel.from_pretrained(model_path, trust_remote_code=True,
                                  attn_implementation='sdpa', torch_dtype=torch.bfloat16)  # sdpa 或 flash_attention_2，不要用 eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(
    model_path, trust_remote_code=True)

多图对话

Python

image1 = Image.open('assets/multi1.png').convert('RGB')
image2 = Image.open('assets/multi2.png').convert('RGB')
question = '比较这两张图片，告诉我它们之间的差异。'

msgs = [{'role': 'user', 'content': [image1, image2, question]}]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

示例图片

示例输出

Plain Text

The vases have different shapes, with the first being rounder and more bulbous. The patterns on the vases are also distinct: the first vase has red designs against a white background, while the second features green and blue floral motifs. Additionally, the neck of the first vase is narrower than that of the second one.

视频理解

加载模型

Python

from PIL import Image
from transformers import AutoModel, AutoTokenizer
import torch

model_path = 'openbmb/MiniCPM-V-4_5'
model = AutoModel.from_pretrained(
    model_path,
    trust_remote_code=True,
    attn_implementation='sdpa',  # sdpa 或 flash_attention_2
    torch_dtype=torch.bfloat16,
)
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(
    model_path, trust_remote_code=True)

视频编码函数

Python

from decord import VideoReader, cpu

MAX_NUM_FRAMES = 64  # 显存吃紧时调小


def encode_video(video_path):
    def uniform_sample(l, n):
        gap = len(l) / n
        idxs = [int(i * gap + gap / 2) for i in range(n)]
        return [l[i] for i in idxs]

    vr = VideoReader(video_path, ctx=cpu(0))
    sample_fps = round(vr.get_avg_fps() / 1)  # 帧率
    frame_idx = [i for i in range(0, len(vr), sample_fps)]
    if len(frame_idx) > MAX_NUM_FRAMES:
        frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES)
    frames = vr.get_batch(frame_idx).asnumpy()
    frames = [Image.fromarray(v.astype('uint8')) for v in frames]
    print('帧数:', len(frames))
    return frames

视频对话

Python

video_path = "assets/badminton.mp4"
frames = encode_video(video_path)
question = "描述这段视频"
msg = [
    {'role': 'user', 'content': frames + [question]},
]

# 视频解码参数
params = {}
params["use_image_id"] = False
# 显存不足且分辨率 > 448*448 时设为 1
params["max_slice_nums"] = 2

res = model.chat(
    image=None,
    msgs=msg,
    tokenizer=tokenizer,
    **params
)

print("Response:")
print(res)

示例输出

Plain Text

num frames: 21
Response:
The video captures a dynamic badminton match between two skilled players, one in yellow and the other in red. The game takes place on an indoor court with green flooring and white lines marking its boundaries. Throughout the footage, we see both athletes displaying impressive footwork as they move swiftly across the playing area to return shots.

At various points, the player wearing yellow is seen executing powerful smashes from different locations around the court, including near the net where he successfully hits his opponent out of bounds twice. In response, the competitor dressed in red demonstrates agility by diving repeatedly onto the floor after failing to retrieve these low shots made close to the ground or driven straight towards him at high speed.

As their rally progresses, each participant showcases quick reflexes and strategic positioning during exchanges that span diagonally across multiple corners while maintaining control over ball placement along sidelines marked by distinct pink borders surrounding them. Their movements are fluid yet intense – reflecting competitive spirit typical for professional sports events like this one which appears staged indoors under artificial lighting conditions visible through overhead fixtures illuminating entire venue space filled with spectators seated behind protective glass panels separating audience members safely away from actual play zone itself (which could be either clay or hard surface based on visual cues).

单图（思考模式）

加载思考模型

Python

from PIL import Image
import torch
from transformers import AutoModel, AutoTokenizer

model_path = 'openbmb/MiniCPM-V-4_5'
model = AutoModel.from_pretrained(model_path, trust_remote_code=True,
                                  # sdpa 或 flash_attention_2，不要用 eager
                                  attn_implementation='sdpa', torch_dtype=torch.bfloat16)
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(
    model_path, trust_remote_code=True)

单图对话

Python

image = Image.open('./assets/single.png').convert('RGB')

# 第一轮对话
question = "图片中的地貌是什么？"
msgs = [{'role': 'user', 'content': [image, question]}]

# 启用思考模式
answer = model.chat(
    msgs=msgs,
    image=image,
    tokenizer=tokenizer,
    enable_thinking=True
)
print(answer)

第二轮对话

Python

# 第二轮对话，带上多轮历史上下文
msgs.append({"role": "assistant", "content": [answer]})
msgs.append({"role": "user", "content": [
            "去这里旅行需要注意什么？"]})

answer = model.chat(
    msgs=msgs,
    image=None,
    tokenizer=tokenizer,
    enable_thinking=True
)
print(answer)

示例图片

示例输出

第一轮

Plain Text

<think>
So, let me analyze the image. The scene shows a long stone staircase winding through lush green mountains, with a pavilion at the top. This looks like a famous mountain path in China. The architectural style and the mountain path are characteristic of Mount Heng (also known as Songshan), which is part of the Hunan province. Wait, or maybe Mount Tai? No, Mount Heng's paths are known for steep steps. Wait, the image might be of the "Heng Mountain's ancient paths" or similar. Wait, the structure at the top is a pavilion, and the winding path through the mountains with such greenery... Let me think. Mount Heng (Hengshan) is one of the Five Sacred Mountains of China, and it has famous ancient paths. Alternatively, maybe Mount Lu? No, the landscape here seems more like Heng Mountain. Wait, the question is asking for the location. Since the image is of a mountain path with traditional Chinese architecture, and the steep steps between mountains, this is likely Mount Heng (Hengshan) in Hunan Province, China. Wait, but I should confirm. Another possibility is Mount Song (Songshan), but Songshan's paths are also steep. Wait, the pavilion at the top—maybe the South Sky Pavilion? Wait, maybe the image is of the "Tianping Steps" or something similar. But given the typical Chinese mountain scenery with such paths, the most famous is Mount Heng. Alternatively, maybe Mount Tai? No, Mount Tai's paths are different. Wait, the greenery and the steep stone steps winding up the mountain suggest this is a well-known sacred mountain path in China. After considering, the location is likely Mount Heng (Hengshan), Hunan Province, China.
</think>

To determine the location, analyze the image: it shows a steep, winding stone path through lush green mountains with traditional Chinese architecture (a pavilion at the top). This landscape is characteristic of Mount Heng (Hengshan), one of China's Five Sacred Mountains, known for its ancient stone steps and mountainous scenery. Thus, the place is located in Hunan Province, China.
Ultimately, the answer is \boxed{Mount Heng (Hengshan), Hunan Province, China}.

第二轮

Plain Text

<think>
So, let's think about what travelers should pay attention to when visiting Mount Heng. First, consider the terrain: the stone steps are steep and may be slippery, so wear comfortable, sturdy shoes and be cautious while climbing. The area is mountainous, so weather can change quickly—bring layers and rain gear. Also, respect local customs and rules, especially around the temples and pavilions. Be prepared for physical exertion since the ascent is long. Maybe also check opening hours and plan accordingly. Plus, the natural beauty is stunning, so allow time to enjoy the scenery.
Ultimately, the answer should cover safety, weather, physical readiness, and cultural respect.
</think>

To determine the location, observe the image's features: a steep stone pathway winding through lush green mountains, accompanied by traditional Chinese architectural elements (such as a pavilion at the top). This landscape is characteristic of **Mount Heng (Hengshan)**, one of China's Five Sacred Mountains, renowned for its ancient stone steps and mountainous terrain.

For precautions when traveling here:
- Wear sturdy, comfortable footwear due to the steep, uneven stone steps.
- Prepare for physical exertion, as the ascent is long and challenging.
- Check weather conditions in advance (mountains can have sudden weather changes) and pack appropriate clothing (layers for temperature shifts, rain gear if necessary).
- Respect local customs and the natural environment (e.g., avoid littering, follow rules around temples/pavilions).
- Plan your visit to allow time for the ascent/descent and to appreciate the scenic views.

Ultimately, the answer is \boxed{Mount Heng (Hengshan), Hunan Province, China}.

多图（思考模式）

加载模型

Python

from PIL import Image
import torch
from transformers import AutoModel, AutoTokenizer

model_path = 'openbmb/MiniCPM-V-4_5'
model = AutoModel.from_pretrained(model_path, trust_remote_code=True,
                                  attn_implementation='sdpa', torch_dtype=torch.bfloat16)  # sdpa 或 flash_attention_2，不要用 eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(
    model_path, trust_remote_code=True)

多图对话

Python

image1 = Image.open('assets/multi1.png').convert('RGB')
image2 = Image.open('assets/multi2.png').convert('RGB')
question = '比较这两张图片，告诉我它们之间的差异。'
# question = 'Compare the two images, tell me about the differences between them.'

msgs = [{'role': 'user', 'content': [image1, image2, question]}]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer,
    enable_thinking=True
)
print(answer)

第二轮对话

Python

# 第二轮对话，带上多轮历史上下文
msgs.append({"role": "assistant", "content": [answer]})
msgs.append({"role": "user", "content": ["告诉我它们的相似之处。"]})
# msgs.append({"role": "user", "content": ["Tell me about the similarities between them."]})
answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer,
    enable_thinking=True
)
print(answer)

示例图片

示例输出

中文回答

第一轮

Plain Text

<think>
那么，让我们从分析每个花瓶的细节开始。首先看第一个花瓶（the first image）：它是一个白色底色的葫芦形花瓶，上面有红色的图案。图案包括人物和云朵，可能还有植物或花卉元素。形状是经典的葫芦形，上下两个球体。

第二个花瓶（the second image）：也是葫芦形，但颜色和图案不同。它有绿色、蓝色、红色和橙色的多彩图案。上半部分有绿色图案，带有红色和橙色的圆形元素。下半部分有绿色藤蔓和花卉设计，更复杂。颜色更丰富，图案种类更多。

所以不同点包括：颜色（第一个是单一红色，第二个是多种颜色：绿、蓝、红、橙）、图案主题（第一个有红色人物和云朵，第二个有绿色藤蔓、花卉和不同形状）、整体设计风格（第一个更简洁的红色图案，第二个更多彩且复杂的图案）。
</think>

要比较这两张图片，可以观察以下不同之处：

1. **颜色方案**：第一个花瓶（the first image）主要是白色底色，上面有红色图案。第二个花瓶（the second image）有绿色、蓝色、红色和橙色等多种颜色的图案。

2. **图案设计**：第一个花瓶的图案包括红色的人物、云朵和植物等元素。第二个花瓶的图案包括绿色藤蔓、花卉、几何形状（如圆形）和不同的装饰元素，整体设计更复杂且色彩丰富。

3. **视觉风格**：第一个花瓶的图案更简约、单色（红色在白色上），而第二个花瓶的图案更复杂、多元（多种颜色和更精致的细节）。


简而言之，不同之处在于颜色方案（第一个是红色，第二个是多种颜色）、图案主题（第一个有红色人物/云朵，第二个有绿色藤蔓/花卉/几何形状）以及整体设计风格（第一个更简约，第二个更复杂）。

第二轮

Plain Text

<think>
明白了，让我们找出它们的相似之处。首先，观察两个花瓶的形状。两者的形状都是双球体（gourd-shaped），由一个较细的上部（neck）和一个较宽的底部（bulbous body）组成。颜色方案？等等，第一个是白色底色上有红色图案，第二个是白色底色上有多种颜色的图案，但它们都是白色底色。等等，是形状吗？是的，形状都是"葫芦形"或双球体花瓶。图案的分布方式？两者都有复杂的装饰图案，覆盖了花瓶的表面。材质？看起来都是陶瓷，可能是瓷器。等等，形状：都是gourd-shaped（双球体），一个较细的上部和一个较宽的底部。颜色方面，都是白色底色，但第一个是红色图案，第二个是多种颜色。但它们都具有双球体形状。图案：两者都有装饰图案，覆盖了花瓶的表面。材质：都是陶瓷。总结相似点：都具有双球体（葫芦形）形状，都有装饰图案，材质可能是瓷器（或类似的陶瓷），都是花瓶。让我们再确认一下。形状：两者的形状都是gourd-shaped（双球体），一个较细的上部和一个较宽的底部。颜色：都是白色底色，尽管第一个是红色图案，第二个是多种颜色，但底色都是白色。图案：两者都有复杂的装饰图案，覆盖了花瓶的表面。材质：看起来都是陶瓷（瓷器）。

因此，它们的相似之处：两者都是双球体花瓶（gourd-shaped），都有装饰图案，材质可能是瓷器或类似的陶瓷，形状都具有类似的葫芦形结构，都用于装饰（花瓶）。
</think>

它们的相似之处：
- 形状：两者都是双球体（葫芦形）花瓶，由一个较细的上部和一个较宽的底部组成。
- 材质：都看起来是陶瓷（可能是瓷器），有白色底色。
- 功能：两者都作为装饰花瓶使用。
- 装饰：都具有复杂的表面图案，虽然图案设计和颜色不同，但都覆盖了花瓶的表面。

English Answer

First Round

Plain Text

<think>
So, let's analyze the two images. First, look at the colors: the first vase (the first image) is red and white, while the second one (the second image) is green, blue, and maybe other colors. Then check the patterns: the first has red cloud-like designs and possibly figures, the second has green floral patterns and different motifs. Also, the shapes might be similar (gourd-shaped) but the decoration styles are different. Wait, the first image's vase has red on white, the second uses green, blue, red, and other colors. The patterns are different too. Let me list the differences:

1. Color scheme: the first image is predominantly red on white, the second is green, blue, and multicolor with floral and geometric patterns.
2. Design patterns: the first has cloud motifs and maybe mythical figures in red, the second has floral designs with green vines, blue borders, and different symbols.
3. Overall style: the first appears to be a classic red-decorated porcelain, the second has a more colorful and possibly multi-colored palette with intricate floral arrangements.
</think>

To compare the two images, we analyze the following differences:

1. **Color Scheme**:
   - The first vase  features a monochromatic red design on a white background.
   - The second vase  uses a multicolored palette, including green, blue, and red, with intricate floral and geometric patterns.

2. **Design Motifs**:
   - The first vase displays red cloud-like patterns and what appears to be a central figure (possibly a mythical or historical character) surrounded by red clouds.
   - The second vase has green vine-like patterns, blue decorative elements, and multicolored floral designs (e.g., red, orange flowers with green leaves) arranged in vertical bands.

3. **Overall Aesthetic Style**:
   - The first vase has a more traditional, single-color red-and-white aesthetic typical of certain classical porcelain styles.
   - The second vase exhibits a vibrant, multi-color decorative style with complex floral and symbolic motifs, reflecting a different artistic tradition.


These differences highlight variations in color, pattern complexity, and design themes between the two vases.

Second Round

Plain Text

<think>
To identify the similarities between the two vases, let's analyze their shared features:

1. **Shape and Form**:
   Both vases have a distinctive double-gourd (or gourd-shaped) silhouette, with a narrower neck and a bulbous, rounded body that tapers into a smaller upper section. This classic gourd shape is a common design element in traditional porcelain.

2. **Material and Finish**:
   Both appear to be made of porcelain, characterized by their smooth, glossy white backgrounds. This indicates a high-quality ceramic material typical of fine Chinese vases.

3. **Decorative Style**:
   Each vase features intricate, hand-painted patterns. The first vase uses a red monochromatic design, while the second uses multiple colors. Despite the difference in color schemes, both vases exhibit detailed, artisanal decoration, showcasing skilled craftsmanship.

4. **Cultural Context**:
   The gourd shape and elaborate painted designs suggest both are influenced by traditional Chinese ceramic art, where such forms and decorative themes are prevalent.

These similarities highlight their shared cultural roots, material quality, and artistic tradition, even as they differ in color and specific motifs.

OCR

加载模型

Python

from PIL import Image
import torch
from transformers import AutoModel, AutoTokenizer

model_path = 'openbmb/MiniCPM-V-4_5'
model = AutoModel.from_pretrained(model_path, trust_remote_code=True,
                                  attn_implementation='sdpa', torch_dtype=torch.bfloat16)  # sdpa 或 flash_attention_2，不要用 eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(
    model_path, trust_remote_code=True)

使用示例

Python

ocr_image = Image.open('./assets/ocr.png').convert('RGB')

question = "图中的文字是什么？"
msgs = [{'role': 'user', 'content': [ocr_image, question]}]

answer = model.chat(
    msgs=msgs,
    image=None,
    tokenizer=tokenizer
)
print(answer)

示例图片

示例输出

Plain Text

2B小钢炮超越 Mistral-7B，面壁发布端侧旗舰 MiniCPM
2024年03月04

以2B规模，在主流榜单多重超越 Mistral-7B，越级比肩 Llama2-13B，乃至更大的几十倍规模模型？

2月1日，旗舰端侧模型「面壁MiniCPM」开源发布，凭借其强有力的性能与极致低成本，堪称大模型里的小钢炮。

面壁 MiniCPM 是端侧模型，更是旗舰模型：

它以小博大，不仅有惊人越级表现，还在堪称最接近用户体感的 MT-Bench 上，表现优异；

它全面开挂，开启端侧部署多模态能力的先河，能说会看，还具备编程能力；

它还是能效之王，极其高效、极低成本，是面壁团队一贯「高效」技术路线成果与「模型沙盒」实验技术创新的集大成者。

面壁 MiniCPM
性能小钢炮

PDF 解析

加载模型

Python

import torch
from transformers import AutoModel, AutoTokenizer

model_path = 'openbmb/MiniCPM-V-4_5'
model = AutoModel.from_pretrained(model_path, trust_remote_code=True,
                                  attn_implementation='sdpa', torch_dtype=torch.bfloat16)  # sdpa 或 flash_attention_2，不要用 eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(
    model_path, trust_remote_code=True)

PDF 转图片

Python

#! 需要先安装 poppler-utils
# sudo apt-get update
# sudo apt-get install poppler-utils
from pdf2image import convert_from_path


def pdf_to_images(pdf_path, dpi=200):
    images = convert_from_path(pdf_path, dpi=dpi)
    images = [image.convert('RGB') for image in images]
    return images

使用示例

Python

prompt = """
You are an OCR assistant. Your task is to identify and extract all visible text from the image provided. Preserve the original formatting as closely as possible, including:

- Line breaks and paragraphs
- Headings and subheadings
- Any tables, lists, bullet points, or numbered items
- Special characters, spacing, and alignment

Output strictly the extracted text in Markdown format, reflecting the layout and structure of the original image. Do not add commentary, interpretation, or summarization—only return the raw text content with its formatting.
"""

contents = [prompt]
contents.extend(pdf_to_images('assets/parse.pdf'))
msgs = [{'role': 'user', 'content': contents}]

answer = model.chat(msgs=msgs, image=None, tokenizer=tokenizer)
print(answer)

示例 PDF

示例输出

Plain Text

For chatbot demo:

pip install -r requirements_o2.6.txt
python web_demos/minicpm-o_2.6/chatbot_web_demo_o2.6.py

Open http://localhost:8000/ in browser and enjoy the vision mode chatbot.

Inference

Model Zoo

| Model | Device | Memory | Description |
|---|---|---|---|
| MiniCPM-o 2.6 | GPU | 18 GB | The latest version, achieving GPT-4o level performance for vision, speech and multimodal live streaming on end-side devices. |
| MiniCPM-o 2.6 gguf | CPU | 8 GB | The gguf version, lower memory usage and faster inference. |
| MiniCPM-o 2.6 int4 | GPU | 9 GB | The int4 quantized version, lower GPU memory usage. |
| MiniCPM-V 2.6 | GPU | 17 GB | Strong end-side multimodal performance for single image, multi-image and video understanding. |
| MiniCPM-V 2.6 gguf | CPU | 6 GB | The gguf version, lower memory usage and faster inference. |
| MiniCPM-V 2.6 int4 | GPU | 7 GB | The int4 quantized version, lower GPU memory usage. |

视觉定位

示例代码

Python

import re
from PIL import Image, ImageDraw
import torch
from transformers import AutoModel, AutoTokenizer

def setup_model_and_tokenizer(model_path):
    dtype = torch.bfloat16
    model = AutoModel.from_pretrained(model_path, torch_dtype=dtype, trust_remote_code=True)
    model = model.to(dtype=torch.bfloat16)
    model = model.eval().cuda()
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    return model, tokenizer

def extract_bbox_from_response(response):
    match = re.search(r"<box>([\d\s]+)</box>", response)
    if match:
        bbox_str = match.group(1)
        bbox = list(map(int, bbox_str.strip().split()))
        return bbox
    else:
        raise ValueError("Can't find bbox in response")

def draw_bbox_on_image(image, bbox):
    w, h = image.size
    x1 = int(bbox[0] / 1000 * w)
    y1 = int(bbox[1] / 1000 * h)
    x2 = int(bbox[2] / 1000 * w)
    y2 = int(bbox[3] / 1000 * h)
    draw = ImageDraw.Draw(image)
    draw.rectangle([x1, y1, x2, y2], outline="red", width=4)
    return image

def model_infer_and_draw(img_path, question, model, tokenizer):
    image = Image.open(img_path)
    msgs = [
        {'role': 'user', 'content': [question, image]},
    ]
    with torch.inference_mode():
        res = model.chat(
            image=None,
            msgs=msgs,
            tokenizer=tokenizer,
            sampling=True,
            max_new_tokens=1024,
            max_inp_length=8192,
            use_image_id=True,
        )
    print("模型输出:", res)
    bbox = extract_bbox_from_response(res)
    image_with_box = draw_bbox_on_image(image, bbox)
    return res, bbox, image_with_box

model_path = 'openbmb/MiniCPM-V-4_5'
img_path = './assets/airplane.jpeg'
question = 'Please provide the bounding box coordinate of the region this sentence describes: <ref>airplane</ref>'

model, tokenizer = setup_model_and_tokenizer(model_path)
res, bbox, image_with_box = model_infer_and_draw(img_path, question, model, tokenizer)

out_path = './assets/airplane_grounding.jpeg'
image_with_box.save(out_path)

效果展示

飞机

Python

model_path = 'openbmb/MiniCPM-V-4_5'
device = 'cuda'
img_path = './assets/airplane.jpeg'
question = 'Please provide the bounding box coordinate of the region this sentence describes: <ref>airplane</ref>'

model, tokenizer = setup_model_and_tokenizer(model_path, device)
res, bbox, image_with_box = model_infer_and_draw(img_path, question, model, tokenizer)

out_path = './assets/airplane-grounding.jpeg'
image_with_box.save(out_path)

原图：

定位后：

哆啦 A 梦

Python

model_path = 'openbmb/MiniCPM-V-4_5'
device = 'cuda'
img_path = './assets/doraemon.jpeg'
question = 'Please provide the bounding box coordinate of the region this sentence describes: <ref>doraemon</ref>'

model, tokenizer = setup_model_and_tokenizer(model_path, device)
res, bbox, image_with_box = model_infer_and_draw(img_path, question, model, tokenizer)

out_path = './assets/doraemon-grounding.jpeg'
image_with_box.save(out_path)

原图：

定位后：

多目标

Python

import re
from PIL import Image, ImageDraw
import torch
from transformers import AutoModel, AutoTokenizer
import json
def setup_model_and_tokenizer(model_path):
    dtype = torch.bfloat16
    model = AutoModel.from_pretrained(model_path, torch_dtype=dtype, trust_remote_code=True)
    model = model.to(dtype=torch.bfloat16)
    model = model.eval().cuda()
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    return model, tokenizer

def extract_bbox_from_json_response(response):
    response_json = json.loads(response)
    bbox_list = []
    name_list = []
    for item in response_json:
        bbox_str = item.get("box", "")
        # 去掉可能存在的 "<box>" / "</box>"
        bbox_str = re.sub(r"</?box>", "", bbox_str)
        name_list.append((item.get("name_en", ""), item.get("name_zh", "")))
        bbox = list(map(int, bbox_str.strip().split()))
        bbox_list.append(bbox)
    return bbox_list, name_list

def draw_multiple_bboxes_on_image(image, bbox_list, name_list):
    w, h = image.size
    draw = ImageDraw.Draw(image)
    for bbox, names in zip(bbox_list, name_list):
        x1 = int(bbox[0] / 1000 * w)
        y1 = int(bbox[1] / 1000 * h)
        x2 = int(bbox[2] / 1000 * w)
        y2 = int(bbox[3] / 1000 * h)
        draw.rectangle([x1, y1, x2, y2], outline="red", width=4)
        draw.text((x1, y1 - 10), f"{names[0]} / {names[1]}", fill="red")
    return image

def model_infer_and_draw(img_path, question, model, tokenizer):
    image = Image.open(img_path)
    msgs = [
        {'role': 'user', 'content': [question, image]},
    ]
    with torch.inference_mode():
        res = model.chat(
            image=None,
            msgs=msgs,
            tokenizer=tokenizer,
            sampling=True,
            max_new_tokens=1024,
            max_inp_length=8192,
            use_image_id=True,
        )
    print("模型输出:", res)
    bbox, name_list = extract_bbox_from_json_response(res)
    image_with_box = draw_multiple_bboxes_on_image(image, bbox, name_list)
    return res, bbox, image_with_box

model_path = 'openbmb/MiniCPM-V-4_5'
img_path = '/cache/liuqilin/draft/example/qwen2v4_5/assets/doraemon.jpg'
question = 'Identify all characters in the image and return their bounding boxes and English name and Chinese name in JSON format, a list of {"name_en": "Doraemon", "name_zh": "哆啦A梦","box", "x1 y1 x2 y2"}'

model, tokenizer = setup_model_and_tokenizer(model_path)
res, bbox, image_with_box = model_infer_and_draw(img_path, question, model, tokenizer)

out_path = './assets/doraemon_multi_grounding.jpg'
image_with_box.save(out_path)

原图：

定位后：