MiniCPM-V 4.6

MiniCPM-V Multimodal Series

Bring the model into your own service: vLLM and SGLang target high-throughput GPU serving, llama.cpp and Ollama target local and on-device runs, while Gradio and iOS are ready-made demo apps. Pick one based on your hardware and scenario; this page covers installation, startup and invocation for each.

MiniCPM-V 4.6 ships as two separate checkpoints; every framework selects one via the model path:

Variant	HuggingFace ID	ModelScope ID
Instruct	openbmb/MiniCPM-V-4.6	OpenBMB/MiniCPM-V-4.6
Thinking	openbmb/MiniCPM-V-4.6-Thinking	OpenBMB/MiniCPM-V-4.6-Thinking

Option	Best for	Notes
vLLM	High-throughput GPU serving	Supported on upstream `main` (PR #43213), no fork needed
SGLang	High-throughput GPU serving	Supported on upstream `main` (PR #24998), no fork needed
llama.cpp	Local CPU / Metal	Supported from release `b9049`; conversion no longer needs the surgery scripts
Ollama	Quick local trial	Built in from v0.30: `ollama run openbmb/minicpm-v4.6`
Gradio	Web demo (image / video chat)	Server plus client
iOS	On-device app (also Android / HarmonyOS)	Built on the `llama.cpp` submodule

Shared prerequisites

v4.6 is registered as MiniCPMV4_6ForConditionalGeneration and requires transformers >= 5.7.0. Unlike v4.5 (which switched modes via enable_thinking), v4.6 deploys the matching checkpoint instead; the vLLM stop_token_ids become [248044, 248046] (v4.5 used [1, 151645]).

vLLM

1. Environment Setup

1.1 Install vLLM

MiniCPM-V 4.6 is supported on the official vLLM main branch since PR #43213 (merged 2026-05-22). No fork is required — install directly from the upstream repo, making sure your checkout is at or after that commit:

Bash

# Create a clean conda environment
conda create -n vllm-v46 python=3.10 -y
conda activate vllm-v46

# Install from upstream main (requires CUDA toolkit; HEAD must include #43213)
git clone https://github.com/vllm-project/vllm.git
cd vllm
MAX_JOBS=6 VLLM_USE_PRECOMPILED=1 pip install --editable . -v

[!TIP] Once a vLLM release containing #43213 is published, you can simply pip install -U vllm and skip the source build. Watch the vLLM Releases page for updates.

For video inference, install the video module:

Bash

pip install vllm[video]

vLLM requires transformers>=5.7.0, in which MiniCPM-V 4.6 has been merged as a standalone architecture (MiniCPMV4_6ForConditionalGeneration).

You can verify the installation with:

Bash

python -c "import vllm, transformers; print('vllm', vllm.__version__, '| transformers', transformers.__version__)"

2. API Service Deployment

2.1 Launch API Service

Bash

vllm serve <model_path> \
  --dtype auto \
  --max-model-len 8192 \
  --api-key token-abc123 \
  --gpu_memory_utilization 0.9 \
  --trust-remote-code \
  --max-num-batched-tokens 8192

Parameter Description:

<model_path>: Local path to MiniCPM-V-4.6, or the HuggingFace ID openbmb/MiniCPM-V-4.6 / openbmb/MiniCPM-V-4.6-Thinking
--api-key: API access key
--max-model-len: Maximum context length. v4.6 supports up to 256K, but start small to fit GPU memory
--gpu_memory_utilization: GPU memory utilization rate

2.2 Image Inference

Python

from openai import OpenAI
import base64

openai_api_key = "token-abc123"  # must match the value passed to `vllm serve`
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)

with open('./assets/airplane.jpeg', 'rb') as file:
    image = "data:image/jpeg;base64," + base64.b64encode(file.read()).decode('utf-8')

chat_response = client.chat.completions.create(
    model="<model_path>",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Please describe this image"},
            {"type": "image_url", "image_url": {"url": image}},
        ],
    }],
    extra_body={
        # v4.6 uses Qwen3.5 backbone with new vocab — note the different stop token IDs
        "stop_token_ids": [248044, 248046]
    }
)

print("Chat response:", chat_response)
print("Chat response content:", chat_response.choices[0].message.content)

2.3 Video Inference

Python

from openai import OpenAI
import base64

openai_api_key = "token-abc123"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)

with open('./videos/video.mp4', 'rb') as video_file:
    video_base64 = base64.b64encode(video_file.read()).decode('utf-8')

chat_response = client.chat.completions.create(
    model="<model_path>",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Please describe this video"},
                {"type": "video_url", "video_url": {"url": f"data:video/mp4;base64,{video_base64}"}},
            ],
        },
    ],
    extra_body={"stop_token_ids": [248044, 248046]}
)

print(chat_response.choices[0].message.content)

v4.6 keeps video pipeline simple: the image processor is invoked frame-by-frame, and the per-video frame budget is controlled by mm_processor_kwargs/video_processor.max_slice_nums. No special video backend is required.

To raise/lower the per-video frame slice budget, pass mm_processor_kwargs:

Python

extra_body = {
    "stop_token_ids": [248044, 248046],
    "mm_processor_kwargs": {"max_slice_nums": 2},
}

2.4 Thinking Mode

If you serve the openbmb/MiniCPM-V-4.6-Thinking checkpoint, the chat template injects a <think> block by default and the assistant returns reasoning followed by </think> then the final answer. You can shortcut the template via chat_template_kwargs:

Python

extra_body = {
    "stop_token_ids": [248044, 248046],
    # disable the leading <think> block on a Thinking model
    "chat_template_kwargs": {"enable_thinking": False},
}

The Instruct checkpoint never emits <think> blocks — pick the right checkpoint instead of toggling at request time.

2.5 Multi-turn Conversation

Launch Parameter Configuration

For multi-image / multi-video conversations, raise the --limit-mm-per-prompt budget at launch:

Bash

# allow up to 3 videos per request
vllm serve <model_path> --dtype auto --max-model-len 16384 --api-key token-abc123 \
  --gpu_memory_utilization 0.9 --trust-remote-code \
  --limit-mm-per-prompt '{"video": 3}'

Bash

# mixed images + video
vllm serve <model_path> --dtype auto --max-model-len 16384 --api-key token-abc123 \
  --gpu_memory_utilization 0.9 --trust-remote-code \
  --limit-mm-per-prompt '{"image": 5, "video": 2}'

Multi-turn Conversation Example

Python

from openai import OpenAI
import base64
import mimetypes
import os

openai_api_key = "token-abc123"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)

messages = [{"role": "system", "content": "You are a helpful assistant."}]

def file_to_base64(file_path):
    with open(file_path, 'rb') as f:
        return base64.b64encode(f.read()).decode('utf-8')

def get_mime_type(file_path):
    mime, _ = mimetypes.guess_type(file_path)
    return mime or 'application/octet-stream'

def build_file_content(file_path):
    mime_type = get_mime_type(file_path)
    base64_data = file_to_base64(file_path)
    url = f"data:{mime_type};base64,{base64_data}"
    if mime_type.startswith("image/"):
        return {"type": "image_url", "image_url": {"url": url}}
    elif mime_type.startswith("video/"):
        return {"type": "video_url", "video_url": {"url": url}}
    else:
        print(f"Unsupported file type: {mime_type}")
        return None

while True:
    user_text = input("Please enter your question (type 'exit' to quit): ")
    if user_text.strip().lower() == "exit":
        break

    content = [{"type": "text", "text": user_text}]

    upload_file = input("Upload a file? (y/n): ").strip().lower() == 'y'
    if upload_file:
        file_path = input("Please enter file path: ").strip()
        if os.path.exists(file_path):
            file_content = build_file_content(file_path)
            if file_content:
                content.append(file_content)
        else:
            print("File path does not exist, skipping file upload.")

    messages.append({"role": "user", "content": content})

    chat_response = client.chat.completions.create(
        model="<model_path>",
        messages=messages,
        extra_body={"stop_token_ids": [248044, 248046]},
    )

    ai_message = chat_response.choices[0].message
    print("MiniCPM-V 4.6:", ai_message.content)

    messages.append({"role": "assistant", "content": ai_message.content})

3. Offline Inference

Python

from transformers import AutoProcessor
from PIL import Image
from vllm import LLM, SamplingParams

MODEL_NAME = "<model_path>"
# Or directly:
# MODEL_NAME = "openbmb/MiniCPM-V-4.6"

image = Image.open("./assets/airplane.jpeg").convert("RGB")
processor = AutoProcessor.from_pretrained(MODEL_NAME)

llm = LLM(
    model=MODEL_NAME,
    max_model_len=8192,
    trust_remote_code=True,
    disable_mm_preprocessor_cache=True,
    limit_mm_per_prompt={"image": 5},
)

messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "Please describe the content of this image"},
    ],
}]

prompt = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image,
        # for multi-image: "image": [image1, image2]
    },
}

sampling_params = SamplingParams(
    stop_token_ids=[248044, 248046],
    temperature=0.7,
    top_p=0.8,
    max_tokens=1024,
)

outputs = llm.generate(inputs, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

Notes

Model Path: replace <model_path> with your local path or one of openbmb/MiniCPM-V-4.6 / openbmb/MiniCPM-V-4.6-Thinking.
Stop Tokens: v4.6 uses the Qwen3.5 vocabulary; the correct stop_token_ids are [248044, 248046] (v4.5 used [1, 151645]).
API Key: ensure the key passed to vllm serve matches the client.
Memory: tune --gpu_memory_utilization, --max-model-len, and --max-num-batched-tokens for your hardware. v4.6 supports up to 256K context, but you typically don't need it.
Multimodal Limits: set --limit-mm-per-prompt for multi-image / multi-video sessions.

SGLang

[!NOTE] MiniCPM-V 4.6 is supported on the official SGLang main branch since PR #24998 (merged 2026-05-12). No fork is required — install directly from upstream, making sure your checkout is at or after that commit.

MiniCPM-V 4.6 is registered in transformers>=5.7.0 as a standalone architecture (MiniCPMV4_6ForConditionalGeneration); the SGLang adapter follows that layout.

MiniCPM-V 4.6 ships as two checkpoints:

Instruct — openbmb/MiniCPM-V-4.6
Thinking — openbmb/MiniCPM-V-4.6-Thinking

1. Installing SGLang

Install SGLang from upstream main

Until a tagged SGLang release ships with #24998, install from upstream main (watch SGLang Releases; once a release includes that PR, pip install -U "sglang[all]" is enough):

Bash

git clone https://github.com/sgl-project/sglang.git
cd sglang

pip install --upgrade pip
pip install -e "python[all]"

transformers>=5.7.0 is installed automatically — this in turn requires a recent PyTorch (≥ 2.6 at the time of writing). Verify the resolved versions match what FlashInfer needs before installing FlashInfer below:

Bash

python -c "import torch, transformers; print('torch', torch.__version__, '| cuda', torch.version.cuda, '| transformers', transformers.__version__)"

Installing FlashInfer (optional but recommended)

[!IMPORTANT] FlashInfer wheels are pinned to a specific (torch, cuda) combo. Pick the wheel index that matches the torch + CUDA you just verified above — don't blindly copy a cu121/torch2.4 URL, that will silently downgrade torch and break the SGLang / transformers install.

The general index lives at <https://flashinfer.ai/whl/>. Pick the directory matching your environment, for example:

Your torch / CUDA	Index URL
torch 2.6 + CUDA 12.4	<https://flashinfer.ai/whl/cu124/torch2.6/>
torch 2.6 + CUDA 12.6	<https://flashinfer.ai/whl/cu126/torch2.6/>
torch 2.7 + CUDA 12.8	<https://flashinfer.ai/whl/cu128/torch2.7/>

Then install via either:

Bash

# Method 1 — pip from the right index (slow / blocked in CN)
pip install flashinfer-python -i <index URL from table above>

# Method 2 — download the matching wheel manually
#   1) Open the index URL in a browser, find a wheel that matches your
#      python version (cp310 / cp311 / ...) and platform (linux_x86_64 / win_amd64)
#   2) pip install <downloaded-wheel.whl>

For everything else (Docker images, CPU-only fallback, etc.) see the official SGLang installation docs.

2. Launching the Inference Server

By default the server downloads weights from the HuggingFace Hub:

Bash

python -m sglang.launch_server --model-path openbmb/MiniCPM-V-4.6 --port 30000 --trust-remote-code --dtype bfloat16

Or specify a local path:

Bash

python -m sglang.launch_server --model-path /your/local/MiniCPM-V-4.6 --port 30000 --trust-remote-code --dtype bfloat16

To serve the Thinking variant, swap the model id:

Bash

python -m sglang.launch_server --model-path openbmb/MiniCPM-V-4.6-Thinking --port 30000 --trust-remote-code --dtype bfloat16

3. Calling the Service

Bash / curl:

Bash

curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-V-4.6",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "What is in this image?"},
          {
            "type": "image_url",
            "image_url": {
              "url": "https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/inference/assets/airplane.jpeg?raw=true"
            }
          }
        ]
      }
    ],
    "max_tokens": 300,
    "top_k": -1
  }'

Python (OpenAI client):

Python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="MiniCPM-V-4.6",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/inference/assets/airplane.jpeg?raw=true",
                    },
                },
            ],
        }
    ],
    max_tokens=300,
    extra_body={"top_k": -1, "stop_token_ids": [248044, 248046]},
)

print(response.choices[0].message.content)

If image_url is not reachable from your machine, replace it with a local path / base64 data URL.

v4.6 uses the Qwen3.5 vocabulary — pass stop_token_ids = [248044, 248046] if you observe the model continuing past the answer.

For more invocation patterns, see the SGLang documentation.

llama.cpp

[!NOTE] MiniCPM-V 4.6 support has been merged into the official llama.cpp (PR #22529) and is included starting from release b9049.

Compared to v4.5, v4.6 reworks the vision tower (the resampler is replaced by a new merger structure for better ViT efficiency), and GGUF conversion is now folded into the standard convert_hf_to_gguf.py flow — no model-specific surgery script is needed.

1. Build llama.cpp

Clone the upstream repository (require commit on/after release b9049):

Bash

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

Build with CMake (see build docs for details):

CPU / Metal:

Bash

cmake -B build
cmake --build build --config Release

CUDA:

Bash

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

2. GGUF files

Option 1: Download official GGUF files

Download the language model file (e.g., MiniCPM-V-4.6-Q4_K_M.gguf) and the vision projector (mmproj-MiniCPM-V-4.6-F16.gguf) from:

HuggingFace: <https://huggingface.co/openbmb/MiniCPM-V-4.6-gguf>
ModelScope: <https://modelscope.cn/models/OpenBMB/MiniCPM-V-4.6-gguf>

The Thinking variant is published separately at:

<https://huggingface.co/openbmb/MiniCPM-V-4.6-Thinking-gguf>

Option 2: Convert from PyTorch model

Download the PyTorch checkpoint:

HuggingFace: <https://huggingface.co/openbmb/MiniCPM-V-4.6> (or MiniCPM-V-4.6-Thinking)
ModelScope: <https://modelscope.cn/models/OpenBMB/MiniCPM-V-4.6>

Run the standard convert_hf_to_gguf.py from the llama.cpp repo:

Bash

# 1) Convert the language model + vision merger to GGUF
python ./convert_hf_to_gguf.py /path/to/MiniCPM-V-4.6 \
    --outfile /path/to/MiniCPM-V-4.6/MiniCPM-V-4.6-F16.gguf \
    --outtype f16

# 2) Convert the vision projector (mmproj)
python ./convert_hf_to_gguf.py /path/to/MiniCPM-V-4.6 \
    --mmproj \
    --outfile /path/to/MiniCPM-V-4.6/mmproj-MiniCPM-V-4.6-F16.gguf

convert_hf_to_gguf.py autodetects MiniCPMV4_6ForConditionalGeneration from config.json and emits both the LM and the vision tower.

v4.6 no longer needs the legacy-models/minicpmv-surgery.py + minicpmv-convert-image-encoder-to-gguf.py two-step. If you're following an older v4.5 guide, ignore those scripts here.

3. Model Inference

[!IMPORTANT] Recent llama.cpp builds (after PR #20606) default --reasoning to auto, which enables thinking from the chat template. The v4.6 Instruct checkpoint does not emit <think> blocks, but its template enables thinking by default, producing broken Instruct output. Always pass --reasoning off explicitly on Instruct inference commands. For the Thinking checkpoint, leave it as default or pass --reasoning on.

Bash

cd build/bin/

# F16 weights
./llama-mtmd-cli \
    -m  /path/to/MiniCPM-V-4.6/MiniCPM-V-4.6-F16.gguf \
    --mmproj /path/to/MiniCPM-V-4.6/mmproj-MiniCPM-V-4.6-F16.gguf \
    -c 8192 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 \
    --reasoning off \
    --image xx.jpg -p "What is in the image?"

# Quantized INT4 weights
./llama-mtmd-cli \
    -m  /path/to/MiniCPM-V-4.6/MiniCPM-V-4.6-Q4_K_M.gguf \
    --mmproj /path/to/MiniCPM-V-4.6/mmproj-MiniCPM-V-4.6-F16.gguf \
    -c 8192 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 \
    --reasoning off \
    --image xx.jpg -p "What is in the image?"

# Interactive mode
./llama-mtmd-cli \
    -m  /path/to/MiniCPM-V-4.6/MiniCPM-V-4.6-Q4_K_M.gguf \
    --mmproj /path/to/MiniCPM-V-4.6/mmproj-MiniCPM-V-4.6-F16.gguf \
    -c 8192 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 \
    --reasoning off \
    --image xx.jpg -i

If you're running the Thinking checkpoint, you can control the reasoning budget through --jinja + --reasoning-budget (--reasoning defaults to auto, which enables thinking for that checkpoint automatically):

Bash

# Allow unlimited thinking (Thinking model)
./llama-mtmd-cli \
    -m  /path/to/MiniCPM-V-4.6-Thinking/MiniCPM-V-4.6-Thinking-Q4_K_M.gguf \
    --mmproj /path/to/MiniCPM-V-4.6-Thinking/mmproj-MiniCPM-V-4.6-Thinking-F16.gguf \
    -c 8192 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 \
    --image xx.jpg --jinja --reasoning-budget -1 -p "What is in the image?"

# Skip the leading <think> block on a Thinking model
./llama-mtmd-cli \
    -m  /path/to/MiniCPM-V-4.6-Thinking/MiniCPM-V-4.6-Thinking-Q4_K_M.gguf \
    --mmproj /path/to/MiniCPM-V-4.6-Thinking/mmproj-MiniCPM-V-4.6-Thinking-F16.gguf \
    -c 8192 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 \
    --image xx.jpg --jinja --reasoning-budget 0 -p "What is in the image?"

The Instruct checkpoint has no <think> block, so --reasoning-budget is a no-op there.

Argument Reference:

Argument	Description
`-m, --model`	Path to the language model
`--mmproj`	Path to the vision projector
`--image`	Path to the input image
`-p, --prompt`	The prompt
`-c, --ctx-size`	Maximum context size
`-rea, --reasoning [on\|off\|auto]`	Enable / disable thinking. Default `auto` (read from chat template). Must be `off` for Instruct; can stay default or be `on` for Thinking
`--reasoning-budget`	Maximum tokens used for reasoning (`-1` unlimited, `0` immediate end); only meaningful for the Thinking checkpoint
`--jinja`	Use the model's Jinja chat template (enabled by default in recent llama.cpp)

Ollama

[!NOTE] MiniCPM-V 4.6 is officially supported by Ollama starting from v0.30. Please make sure your Ollama version is 0.30 or newer; older releases do not include the model and will fail to load it.

1. Install Ollama

macOS: download from <https://ollama.com/download/Ollama.dmg>.
Windows: download from <https://ollama.com/download/OllamaSetup.exe>.
Linux: curl -fsSL https://ollama.com/install.sh | sh, or follow the Linux install guide.
Docker: official Ollama Docker image ollama/ollama is on Docker Hub.

Verify your version is at least 0.30:

Bash

ollama --version

2. Quick Start

Pull and run the official OpenBMB model from the Ollama registry (<https://ollama.com/openbmb/minicpm-v4.6>):

Bash

ollama run openbmb/minicpm-v4.6
# or, for the Thinking variant
ollama run openbmb/minicpm-v4.6-thinking

Command line

Separate the input prompt and the image path with a space:

Plain Text

What is in the picture? xx.jpg

API

Python

import base64, requests

with open(image_path, 'rb') as image_file:
    encoded_string = base64.b64encode(image_file.read()).decode('utf-8')
    data = {
        "model": "openbmb/minicpm-v4.6",
        "prompt": query,
        "stream": False,
        "images": [encoded_string],  # the list can hold multiple base64-encoded images
    }
    url = "http://localhost:11434/api/generate"
    response = requests.post(url, json=data)

3. Customize model

If you'd like to load a local GGUF (e.g. a custom quantization), follow the steps below.

Download GGUF Model

HuggingFace: <https://huggingface.co/openbmb/MiniCPM-V-4.6-gguf>
ModelScope: <https://modelscope.cn/models/OpenBMB/MiniCPM-V-4.6-gguf>

(Or openbmb/MiniCPM-V-4.6-Thinking-gguf for the Thinking variant.)

Create a ModelFile

Bash

vim minicpmv4.6.Modelfile

ModelFile content:

Plain Text

FROM ./MiniCPM-V-4.6/MiniCPM-V-4.6-Q4_K_M.gguf
FROM ./MiniCPM-V-4.6/mmproj-MiniCPM-V-4.6-F16.gguf

TEMPLATE """{{- if .Messages }}{{- range $i, $_ := .Messages }}{{- $last := eq (len (slice $.Messages $i)) 1 -}}<|im_start|>{{ .Role }}{{ .Content }}{{- if $last }}{{- if (ne .Role "assistant") }}<|im_end|><|im_start|>assistant{{ end }}{{- else }}<|im_end|>{{ end }}{{- end }}{{- else }}{{- if .System }}<|im_start|>system{{ .System }}<|im_end|>{{ end }}{{ if .Prompt }}<|im_start|>user{{ .Prompt }}<|im_end|>{{ end }}<|im_start|>assistant{{ end }}{{ .Response }}{{ if .Response }}<|im_end|>{{ end }}"""

SYSTEM """You are a helpful assistant."""

PARAMETER top_p 0.8
PARAMETER num_ctx 8192
PARAMETER stop ["<|im_start|>", "<|im_end|>"]
PARAMETER temperature 0.7

first FROM	second FROM	num_ctx
Path to the language GGUF	Path to the vision projector GGUF	Maximum context length

Create the Ollama model

Bash

ollama create minicpm-v4.6 -f minicpmv4.6.Modelfile

Run

In a new terminal:

Bash

ollama run minicpm-v4.6

Input prompt

Enter the prompt and the image path separated by a space:

Plain Text

What is in the picture? xx.jpg

Gradio

Single-process Gradio demo for MiniCPM-V 4.6. One process can load BOTH the instruct and thinking checkpoints at once; the "Thinking Mode" toggle in the UI then switches the active model (and flips enable_thinking in the chat template) on the fly.

Unlike the v4.5 demo (which uses a FastAPI server + Gradio client split and the model.chat(...) custom API), v4.6 upstreams to the standard HuggingFace transformers API:

Python

from transformers import AutoProcessor, MiniCPMV4_6ForConditionalGeneration

processor = AutoProcessor.from_pretrained(path)
model     = MiniCPMV4_6ForConditionalGeneration.from_pretrained(path, dtype=torch.bfloat16)

inputs = processor.apply_chat_template(messages, add_generation_prompt=True,
                                       tokenize=True, return_dict=True,
                                       return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=..., do_sample=...)
text = processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)

Streaming is implemented via transformers.TextIteratorStreamer in a background thread so the Gradio UI can yield chunks token-by-token.

Directory layout

Plain Text

gradio/v46/
├── app.py          # single-process Gradio app (loads 1 or 2 checkpoints)
├── start.sh        # tmux launcher (dual / instruct / thinking variants, LB registration)
├── requirements.txt
└── README.md

Models

Variant	Path
instruct	`./minicpm-v-4_6-0420-rlaif-instruct`
thinking	`./minicpm-v-4_6-0420-rlaif-thinking`

Note: config.json for both was patched to include image_token_id: 248056 (the id of the <|image_pad|> special token) — without this the model raises AttributeError: 'bool' object has no attribute 'sum' inside get_placeholder_mask.

Environment

A dedicated v46 conda env was cloned from omni and then:

Bash

conda create --clone omni -n v46 --offline
conda activate v46

PYTHONNOUSERSITE=1 pip install -e ./new-model-addition-MiniCPM-V-4.6 --no-deps
PYTHONNOUSERSITE=1 pip install -U "huggingface_hub>=1.0" "tokenizers>=0.22.0,<=0.23.0" "regex>=2025.10.22" "mistral_common>=1.11.0"

PYTHONNOUSERSITE=1 is required when the host has site-packages that shadow the conda-env transformers and huggingface_hub with incompatible versions.

A small patch was also applied to src/transformers/models/minicpmv4_6/configuration_minicpmv4_6.py in your environment:

Python

# BEFORE:
merge_kernel_size: list[int] = [2, 2]
# AFTER:
from dataclasses import field
merge_kernel_size: list[int] = field(default_factory=lambda: [2, 2])

(Mutable-list defaults are not allowed in Python 3.10+ @dataclass.)

How the Thinking toggle works

Plain Text

  ┌─────────────────────────────────────────────┐
  │  one app.py process  (same CUDA device)     │
  │                                             │
  │   MODELS = {                                │
  │     "instruct":  <MiniCPMV4_6…instruct>     │
  │     "thinking":  <MiniCPMV4_6…thinking>     │
  │   }                                         │
  └─────────────────────────────────────────────┘

  Checkbox OFF  →  variant="instruct",  enable_thinking=False
  Checkbox ON   →  variant="thinking",  enable_thinking=True

When the checkbox is flipped, the chat history is automatically cleared and a toast Switched to 'thinking' model, history cleared. is shown. History is cleared because the two checkpoints produce stylistically different outputs (<think>…</think> vs plain answer) and mixing them in one conversation tends to confuse the model on later turns.

GPU memory: each checkpoint is ≈16 GB in bfloat16, so the dual-model process needs ≈32 GB. Recommended on 80 GB A100/H100. On smaller cards, launch with --variant instruct or --variant thinking to load only one checkpoint.

Launch

A. Quick start — one dual-model instance on a single GPU

Bash

cd ./MiniCPM-o-demo-web/gradio/v46

# dual-model on GPU 7, port 8890, no load balancer
bash start.sh -n 1 --gpu-start 7 --port-base 8890 --no-lb

Browse to http://<host>:8890, flip the "Thinking Mode" checkbox to switch checkpoints.

B. Production — multiple dual-model instances behind a load balancer

Architecture:

Plain Text

                  ┌──────────────────────────────────────────┐
      user ─────▶ │  load_balancer :8121  (ip_hash + SSE)    │
                  └──┬───────────────┬──────────────┬────────┘
                     │               │              │
                     ▼               ▼              ▼
              127.0.0.1:8890  127.0.0.1:8891  127.0.0.1:8892     ← v4.6 app.py (dual)
              (GPU 7)         (GPU 6)         (GPU 5)
                 │               │              │
                 └─── each process holds BOTH instruct + thinking ───

Because every backend serves BOTH variants, there's only one LB pool, and ip_hash session stickiness keeps each user on one backend (so their checkbox state stays consistent).

1) Start the load balancer

Bash

cd ../load_balancer
python load_balancer.py --port 8121 --strategy ip_hash

2) Start the Gradio instances (auto-registers to LB)

Bash

cd ../v46

# 4 dual-model instances on GPU 7,6,5,4 → LB :8121
bash start.sh -n 4 \
    --gpu-start 7 --port-base 8890 \
    --lb-host 127.0.0.1 --lb-port 8121

End users access: http://<host>:8121.

3) Status / stop

Bash

bash start.sh --status
bash start.sh --stop        # also unregisters from LB

C. Single-variant cluster (less GPU memory per process)

If your cards can't fit ~32 GB, deploy the two variants separately. In this mode the "Thinking Mode" toggle only flips enable_thinking (it cannot switch models because only one is loaded), and you need two LB ports so users don't accidentally land on the other model.

Bash

bash start.sh -n 4 --variant instruct  --gpu-start 7 --port-base 8890 --lb-port 8121
bash start.sh -n 4 --variant thinking  --gpu-start 3 --port-base 8900 --lb-port 8122

D. Run app.py manually (no tmux, no LB)

Bash

conda activate v46

# Dual-model (checkbox switches models)
PYTHONNOUSERSITE=1 CUDA_VISIBLE_DEVICES=7 python app.py \
    --instruct_path=minicpm-v-4_6-0420-rlaif-instruct \
    --thinking_path=minicpm-v-4_6-0420-rlaif-thinking \
    --port=8890

# Single-model
PYTHONNOUSERSITE=1 CUDA_VISIBLE_DEVICES=7 python app.py \
    --instruct_path=minicpm-v-4_6-0420-rlaif-instruct \
    --port=8890

Multi-machine deployment

Same as v4.5. Run the LB on the primary machine, then on each worker machine:

Bash

bash start.sh -n K --lb-host <primary-ip> --lb-port 8121 --local-ip <this-ip>

UI features

Upload: single or multiple images, or one video per turn. No mixing video+image (enforced on the backend).
Decode Type: Beam Search (deterministic, num_beams=1) or Sampling.
Thinking Mode: switches the active checkpoint when two are loaded, or flips enable_thinking when only one is loaded. Toggling this clears the chat history to avoid mixing output styles.
Enable Streaming Mode: token-by-token updates via TextIteratorStreamer. Beam Search disables streaming automatically.
Sliders: max_new_tokens, temperature, top_p, top_k.
Regenerate / Clear History / Stop buttons.
<think>…</think> segments are rendered in a distinct blue card above the answer so you can see the model's reasoning in real time.

Known differences from v4.5 demo

Aspect	v4.5	v4.6
Architecture	FastAPI server + Gradio client + LB	Single-process Gradio app (optionally behind LB)
Models per process	1	1 or 2 (dual instruct+thinking)
Model loading	`AutoModel.from_pretrained(trust_remote_code=True)`	`MiniCPMV4_6ForConditionalGeneration.from_pretrained(...)`
Inference	`model.chat(msgs, tokenizer, processor, ...)`	`model.generate(**processor.apply_chat_template(...))`
Video encoding	Client pre-extracts frames → base64 per-frame POST	Processor extracts frames internally from a local path
Streaming	Custom `chat(stream=True)` → SSE over HTTP	`TextIteratorStreamer` in a thread → direct Gradio yield
Thinking mode	`enable_thinking` to `model.chat`	`enable_thinking` to `apply_chat_template` + model switch

iOS

Upstream project: OpenBMB/MiniCPM-V-Apps

README: English · 简体中文

That repository bundles iOS, Android, and HarmonyOS NEXT demos sharing one root llama.cpp git submodule (branch Support-iOS-Demo). This page summarizes the iOS flow; Android / HarmonyOS build steps stay in the upstream README.

Prebuilt installers (TestFlight / APK / HAP): DOWNLOAD.md / DOWNLOAD_zh.md. The sections below are for building from source.

1. Clone repo and submodule

Bash

git clone https://github.com/OpenBMB/MiniCPM-V-Apps.git
cd MiniCPM-V-Apps
git submodule update --init --recursive

2. Open the Xcode project

NOTE: Deploying on a physical iPhone or iPad may require an Apple Developer membership.

Install Xcode:

Xcode from the App Store
Command Line Tools:
Bash
```
xcode-select --install
```
Accept the license:
Bash
```
sudo xcodebuild -license
```

Open MiniCPM-V-demo/MiniCPM-V-demo.xcodeproj in Xcode and let it finish downloading the required dependencies. Then pick a destination device at the top and tap Run.

NOTE: If something fails related to thirdparty/llama.xcframework, build it manually using section 3.

3. Manually build `llama.xcframework`

From the repository root (after submodules are in place):

Bash

cd llama.cpp
./build-xcframework.sh
cp -r ./build-apple/llama.xcframework ../MiniCPM-V-demo/thirdparty

4. GGUF model files for the demos

Upstream currently targets MiniCPM-V 2.6, 4.0, and 4.6. For each version, download a quantized language-model GGUF plus mmproj-model-f16.gguf (the vision projector is kept at f16 since quantizing the visual tower hurts perception quality noticeably more than quantizing the LLM).

Recommended device RAM and total download size (see upstream README hardware section):

Model	LLM params	Recommended quant	Total download	Recommended device RAM
MiniCPM-V 2.6	8B	Q4_K_M	~5.4 GB	≥ 8 GB
MiniCPM-V 4.0	4.1B	Q4_K_M	~2.9 GB	≥ 6 GB
MiniCPM-V 4.6	1.3B	Q4_K_M	~1.6 GB	≥ 6 GB

All three demos default to a 4K context window; KV-cache footprint grows roughly linearly with context, so on a borderline device you may want to lower it.

MiniCPM-V 2.6 — official GGUF

Hugging Face: openbmb/MiniCPM-V-2_6-gguf
ModelScope: OpenBMB/MiniCPM-V-2_6-gguf

Download the language-model file (e.g. ggml-model-Q4_0.gguf) and the vision-model file mmproj-model-f16.gguf.

MiniCPM-V 4.0 — official GGUF

Hugging Face: openbmb/MiniCPM-V-4-gguf
ModelScope: OpenBMB/MiniCPM-V-4-gguf

Download the language-model file (e.g. ggml-model-Q4_K_M.gguf) and the vision-model file mmproj-model-f16.gguf.

MiniCPM-V 4.6 — official GGUF

Hugging Face: openbmb/MiniCPM-V-4.6-gguf
ModelScope: OpenBMB/MiniCPM-V-4.6-gguf

Download the language-model file (e.g. MiniCPM-V-4_6-Q4_K_M.gguf) and the vision-model file mmproj-model-f16.gguf.

5. Convert PyTorch → GGUF (optional)

For server-side conversion recipes in this Cookbook, see:

MiniCPM-V 4.6 GGUF
MiniCPM-V 4.0 GGUF
Additional versions: Cookbook quantization/gguf/

Run conversion commands inside the llama.cpp subtree at the MiniCPM-V-Apps repo root.

vLLM​

1. Environment Setup​

1.1 Install vLLM​

2. API Service Deployment​

2.1 Launch API Service​

2.2 Image Inference​

2.3 Video Inference​

2.4 Thinking Mode​

2.5 Multi-turn Conversation​

Launch Parameter Configuration​

Multi-turn Conversation Example​

3. Offline Inference​

Notes​

SGLang​

1. Installing SGLang​

Install SGLang from upstream main​

Installing FlashInfer (optional but recommended)​

2. Launching the Inference Server​

3. Calling the Service​

llama.cpp​

1. Build llama.cpp​

2. GGUF files​

Option 1: Download official GGUF files​

Option 2: Convert from PyTorch model​

3. Model Inference​

Ollama​

1. Install Ollama​

2. Quick Start​

Command line​

API​

3. Customize model​

Download GGUF Model​

Create a ModelFile​

Create the Ollama model​

Run​

Input prompt​

Gradio​

Directory layout​

Models​

Environment​

How the Thinking toggle works​

Launch​

A. Quick start — one dual-model instance on a single GPU​

B. Production — multiple dual-model instances behind a load balancer​

1) Start the load balancer​

2) Start the Gradio instances (auto-registers to LB)​

3) Status / stop​

C. Single-variant cluster (less GPU memory per process)​

D. Run app.py manually (no tmux, no LB)​

Multi-machine deployment​

UI features​

Known differences from v4.5 demo​

iOS​

1. Clone repo and submodule​

2. Open the Xcode project​

3. Manually build llama.xcframework​

4. GGUF model files for the demos​

MiniCPM-V 2.6 — official GGUF​

MiniCPM-V 4.0 — official GGUF​

MiniCPM-V 4.6 — official GGUF​

5. Convert PyTorch → GGUF (optional)​

vLLM

1. Environment Setup

1.1 Install vLLM

2. API Service Deployment

2.1 Launch API Service

2.2 Image Inference

2.3 Video Inference

2.4 Thinking Mode

2.5 Multi-turn Conversation

Launch Parameter Configuration

Multi-turn Conversation Example

3. Offline Inference

Notes

SGLang

1. Installing SGLang

Install SGLang from upstream main

Installing FlashInfer (optional but recommended)

2. Launching the Inference Server

3. Calling the Service

llama.cpp

1. Build llama.cpp

2. GGUF files

Option 1: Download official GGUF files

Option 2: Convert from PyTorch model

3. Model Inference

Ollama

1. Install Ollama

2. Quick Start

Command line

API

3. Customize model

Download GGUF Model

Create a ModelFile

Create the Ollama model

Run

Input prompt

Gradio

Directory layout

Models

Environment

How the Thinking toggle works

Launch

A. Quick start — one dual-model instance on a single GPU

B. Production — multiple dual-model instances behind a load balancer

1) Start the load balancer

2) Start the Gradio instances (auto-registers to LB)

3) Status / stop

C. Single-variant cluster (less GPU memory per process)

D. Run app.py manually (no tmux, no LB)

Multi-machine deployment

UI features

Known differences from v4.5 demo

iOS

1. Clone repo and submodule

2. Open the Xcode project

3. Manually build `llama.xcframework`

4. GGUF model files for the demos

MiniCPM-V 2.6 — official GGUF

MiniCPM-V 4.0 — official GGUF

MiniCPM-V 4.6 — official GGUF

5. Convert PyTorch → GGUF (optional)