MiniCPM-V 4.6

MiniCPM-V Multimodal Series

Trade a little quality for much lower memory and storage. GGUF suits local and CPU / Metal runs, while BNB and AWQ target GPU inference. Use the official pre-quantized weights, or quantize yourself with the scripts on this page.

Format	Precision	Tool	Best for
GGUF	Q4_K_M and other levels	llama.cpp	Local runs with llama.cpp / Ollama
BNB	int4	bitsandbytes	GPU inference, simplest script
AWQ	4-bit	AutoAWQ	GPU inference, needs calibration data

What changed from v4.5

v4.6 conversion is simpler. The model is registered upstream in transformers>=5.7.0, and llama.cpp's standard convert_hf_to_gguf.py (≥ release b9049) handles both the language model and the vision projector. The legacy minicpmv-surgery.py + minicpmv-convert-image-encoder-to-gguf.py scripts are no longer needed.

GGUF

This guide walks through converting the MiniCPM-V 4.6 PyTorch model to GGUF and quantizing it.

The resulting GGUFs are meant to be used with llama.cpp or ollama.

1. Download the PyTorch model

Pick the variant you want to quantize:

Instruct — HuggingFace: <https://huggingface.co/openbmb/MiniCPM-V-4.6> · ModelScope: <https://modelscope.cn/models/OpenBMB/MiniCPM-V-4.6>
Thinking — HuggingFace: <https://huggingface.co/openbmb/MiniCPM-V-4.6-Thinking> · ModelScope: <https://modelscope.cn/models/OpenBMB/MiniCPM-V-4.6-Thinking>

2. Convert to GGUF

Run from the root of a llama.cpp checkout (release b9049 or later):

Bash

# Step 1 — convert the language model + vision merger to F16 GGUF
python ./convert_hf_to_gguf.py /path/to/MiniCPM-V-4.6 \
    --outfile /path/to/MiniCPM-V-4.6/MiniCPM-V-4.6-F16.gguf \
    --outtype f16

# Step 2 — convert the vision projector (mmproj)
python ./convert_hf_to_gguf.py /path/to/MiniCPM-V-4.6 \
    --mmproj \
    --outfile /path/to/MiniCPM-V-4.6/mmproj-MiniCPM-V-4.6-F16.gguf

convert_hf_to_gguf.py autodetects MiniCPMV4_6ForConditionalGeneration from config.json.

3. Quantize to INT4

Once you have the F16 LM GGUF, quantize it with llama-quantize:

Bash

./llama-quantize \
    /path/to/MiniCPM-V-4.6/MiniCPM-V-4.6-F16.gguf \
    /path/to/MiniCPM-V-4.6/MiniCPM-V-4.6-Q4_K_M.gguf \
    Q4_K_M

Other common quant types you can swap in: Q5_K_M, Q6_K, Q8_0. Pick based on your accuracy vs. memory tradeoff.

The mmproj file is already small — keep it at F16 and pair it with whichever quantization you chose for the LM.

BNB

Support: MiniCPM-V 4.6 / MiniCPM-V 4.5 / MiniCPM-V 4.0 / MiniCPM-V 2.6 / MiniCPM-V 2.5

1. Download the Model

Download the MiniCPM-V-4.6 model from HuggingFace (or openbmb/MiniCPM-V-4.6-Thinking for the think variant) and extract it to a local directory.

2. Quantization Script

The following script loads the original model, quantizes it to 4-bit using bitsandbytes, runs a sanity-check inference, and saves the quantized model.

Requires transformers>=5.7.0 (where MiniCPM-V 4.6 is registered as a standalone architecture).

Python

import os
import time
import torch
from transformers import (
    AutoProcessor,
    AutoModelForImageTextToText,
    BitsAndBytesConfig,
)
from PIL import Image

assert torch.cuda.is_available(), "CUDA is not available, but this code requires a GPU."

device = "cuda"
model_path = "/model/MiniCPM-V-4.6"  # Original model path
save_path = "./model/MiniCPM-V-4.6-int4"  # Path to save quantized model
image_path = "./assets/airplane.jpeg"

# 4-bit quantization configuration
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    load_in_8bit=False,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_storage=torch.uint8,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    llm_int8_enable_fp32_cpu_offload=False,
    llm_int8_has_fp16_weight=False,
    llm_int8_skip_modules=["lm_head"],
    llm_int8_threshold=6.0,
)

processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForImageTextToText.from_pretrained(
    model_path,
    device_map=device,
    quantization_config=quantization_config,
)

image = Image.open(image_path).convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "What is in this picture?"},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

start = time.time()
out_ids = model.generate(**inputs, max_new_tokens=256)
response = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)

print("Output after quantization:", response)
print("Inference time after quantization:", time.time() - start)

# Save the quantized model and tokenizer/processor
os.makedirs(save_path, exist_ok=True)
model.save_pretrained(save_path, safe_serialization=True)
processor.save_pretrained(save_path)

3. Expected Output

After quantization you should see something like:

Plain Text

Output after quantization: The image depicts an Airbus A380-800 aircraft in mid-flight against a clear blue sky. The airplane is predominantly white with a distinctive blue tail fin featuring a red and white logo. ...
Inference time after quantization: ~9 s on a single A100/4090

The quantized model will be saved at save_path and can be reloaded with AutoModelForImageTextToText.from_pretrained(save_path) for further inference or fine-tuning.

To customize the model path, image, or save path, edit the variables at the top of the script.

AWQ

Support: MiniCPM-V 4.6 (Instruct & Thinking) / MiniCPM-V 4.5

The AutoAWQ workflow used by MiniCPM-V is hosted at tc-mb/AutoAWQ (the upstream casper-hansen/AutoAWQ is no longer maintained).

MiniCPM-V 4.6 requires transformers>=5.7.0; the AutoAWQ fork tracks that requirement.

Method 1 — Use the pre-quantized model with vLLM

1. Download the pre-quantized model

Bash

git clone https://huggingface.co/openbmb/MiniCPM-V-4.6-AWQ

(Or openbmb/MiniCPM-V-4.6-Thinking-AWQ for the Thinking variant once published.)

2. Run with vLLM

Python

from PIL import Image
from transformers import AutoProcessor
from vllm import LLM, SamplingParams

MODEL_NAME = "openbmb/MiniCPM-V-4.6-AWQ"
IMAGES = ["image.png"]

image = Image.open(IMAGES[0]).convert("RGB")
processor = AutoProcessor.from_pretrained(MODEL_NAME)

llm = LLM(
    model=MODEL_NAME,
    max_model_len=8192,
    trust_remote_code=True,
    # limit_mm_per_prompt={"image": 5},
)

messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "Please describe the content of this image"},
    ],
}]

prompt = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

sampling_params = SamplingParams(
    stop_token_ids=[248044, 248046],   # v4.6 uses Qwen3.5 vocab
    temperature=0.7,
    top_p=0.8,
    max_tokens=1024,
)

outputs = llm.generate(
    {
        "prompt": prompt,
        "multi_modal_data": {"image": image},
    },
    sampling_params=sampling_params,
)
print(outputs[0].outputs[0].text)

Method 2 — Run the AWQ checkpoint with AutoAWQ directly

1. Download the model

Bash

git clone https://huggingface.co/openbmb/MiniCPM-V-4.6-AWQ

2. Build AutoAWQ from source

Bash

git clone https://github.com/tc-mb/AutoAWQ.git
cd AutoAWQ
pip install -e .

3. Inference script

Python

import torch
from PIL import Image
from transformers import AutoProcessor
from awq import AutoAWQForCausalLM

model_path = "openbmb/MiniCPM-V-4.6-AWQ"
image_path = "./assets/airplane.jpeg"

model = AutoAWQForCausalLM.from_quantized(
    model_path, trust_remote_code=True
).to("cuda")
processor = AutoProcessor.from_pretrained(model_path)

image = Image.open(image_path).convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "What is in this picture?"},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to("cuda")

with torch.inference_mode():
    out_ids = model.generate(**inputs, max_new_tokens=256)

response = processor.decode(
    out_ids[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print("Output:", response)

Method 3 — Quantize the model yourself

1. Download the original model

Bash

git clone https://huggingface.co/openbmb/MiniCPM-V-4.6

2. Build AutoAWQ from source

Bash

git clone https://github.com/tc-mb/AutoAWQ.git
cd AutoAWQ
pip install -e .

3. Quantization script

Python

import os
import shutil
import torch
from datasets import load_dataset
from transformers import AutoTokenizer
from awq import AutoAWQForCausalLM

model_path = "/openbmb/MiniCPM-V-4.6"
quant_path = "/model_quantized/minicpmv4_6_awq"

# AWQ config — 4-bit weights, group size 128, GEMM backend
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",
}

model = AutoAWQForCausalLM.from_pretrained(
    model_path, trust_remote_code=True, torch_dtype=torch.bfloat16
)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)


def copy_files_not_in_B(A_path, B_path):
    """Copy non-weight files from A to B if missing."""
    if not os.path.exists(A_path):
        raise FileNotFoundError(f"The directory {A_path} does not exist.")
    if not os.path.exists(B_path):
        os.makedirs(B_path)

    files_in_A = set(
        f for f in os.listdir(A_path)
        if not (".bin" in f or "safetensors" in f)
    )
    files_in_B = set(os.listdir(B_path))

    for f in files_in_A - files_in_B:
        src = os.path.join(A_path, f)
        dst = os.path.join(B_path, f)
        if os.path.isfile(src):
            shutil.copy2(src, dst)


def load_alpaca():
    data = load_dataset("tatsu-lab/alpaca", split="train")

    def concatenate(x):
        if x["input"] and x["instruction"]:
            msgs = [
                {"role": "system", "content": x["instruction"]},
                {"role": "user", "content": x["input"]},
                {"role": "assistant", "content": x["output"]},
            ]
        elif x["input"]:
            msgs = [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": x["input"]},
                {"role": "assistant", "content": x["output"]},
            ]
        else:
            msgs = [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": x["instruction"]},
                {"role": "assistant", "content": x["output"]},
            ]
        text = tokenizer.apply_chat_template(
            msgs, tokenize=False, add_generation_prompt=True
        )
        return {"text": text}

    return [r["text"] for r in data.map(concatenate)][:1024]


calib_data = load_alpaca()
model.quantize(tokenizer, quant_config=quant_config, calib_data=calib_data)

model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

copy_files_not_in_B(model_path, quant_path)
print(f'Model is quantized and saved at "{quant_path}"')

GGUF​

1. Download the PyTorch model​

2. Convert to GGUF​

3. Quantize to INT4​

BNB​

1. Download the Model​

2. Quantization Script​

3. Expected Output​

AWQ​

Method 1 — Use the pre-quantized model with vLLM​

1. Download the pre-quantized model​

2. Run with vLLM​

Method 2 — Run the AWQ checkpoint with AutoAWQ directly​

1. Download the model​

2. Build AutoAWQ from source​

3. Inference script​

Method 3 — Quantize the model yourself​

1. Download the original model​

2. Build AutoAWQ from source​

3. Quantization script​

GGUF

1. Download the PyTorch model

2. Convert to GGUF

3. Quantize to INT4

BNB

1. Download the Model

2. Quantization Script

3. Expected Output

AWQ

Method 1 — Use the pre-quantized model with vLLM

1. Download the pre-quantized model

2. Run with vLLM

Method 2 — Run the AWQ checkpoint with AutoAWQ directly

1. Download the model

2. Build AutoAWQ from source

3. Inference script

Method 3 — Quantize the model yourself

1. Download the original model

2. Build AutoAWQ from source

3. Quantization script