MiniCPM-V 4.7

MiniCPM-V Multimodal Series

Trade smaller weights for lower GPU memory and storage. GGUF suits local and CPU / Metal runs, BNB / AWQ / GPTQ target GPU inference, while the MoE experts in 35B-A3B require llm-compressor. Use the official pre-quantized weights, or quantize yourself with the scripts on this page.

Format	1B	35B-A3B	Repo suffix	Tool
GGUF	✅	✅	-gguf	llama.cpp
BNB (int4)	✅	Use GGUF / pre-quantized weights instead	-int4	bitsandbytes
AWQ	✅	Use GGUF / pre-quantized weights instead	-AWQ	AutoAWQ
GPTQ	✅	Use GGUF / pre-quantized weights instead	-GPTQ	AutoGPTQ
FP8 / INT4	—	✅	-FP8 / -INT4	llm-compressor

35B-A3B's fused experts

MiniCPM-V 4.7 35B-A3B's MoE experts are stored as fused parameters (mlp.experts.gate_up_proj, mlp.experts.down_proj). bitsandbytes / AutoAWQ / AutoGPTQ only quantize nn.Linear, leaving the experts (about 92% of weights) at bf16 (saved checkpoint still ~60 GB). For usable 4-bit on 35B-A3B, use GGUF or the published -INT4 / -FP8 weights, see FP8 / INT4.

GGUF

Convert the PyTorch model to GGUF and quantize it, producing outputs for llama.cpp or Ollama.

v4.7 uses canvas M-RoPE, introducing new GGUF metadata keys (rope.section_order) and mmproj tensors (v.tok_embd_sep). Please use llama.cpp at or after PR #29416 to reconvert; GGUFs converted with older scripts will compute M-RoPE incorrectly in recent builds.

Pre-built GGUFs are available: openbmb/MiniCPM-V-4.7-1B-gguf / openbmb/MiniCPM-V-4.7-35B-A3B-gguf.

Convert to GGUF

Run from the root of a current (includes PR #29416) llama.cpp checkout:

Bash

# Step 1 — convert the language model + vision merger to F16 GGUF
python ./convert_hf_to_gguf.py /path/to/MiniCPM-V-4.7-1B \
    --outfile /path/to/MiniCPM-V-4.7-1B/MiniCPM-V-4_7-1B-F16.gguf \
    --outtype f16

# Step 2 — convert the vision projector (mmproj)
python ./convert_hf_to_gguf.py /path/to/MiniCPM-V-4.7-1B \
    --mmproj \
    --outfile /path/to/MiniCPM-V-4.7-1B/mmproj-model-f16.gguf

convert_hf_to_gguf.py autodetects MiniCPMV4_7ForConditionalGeneration from config.json.

To export the 4× vision variant: copy the model directory, change "downsample_mode" to "4x" in preprocessor_config.json, then run --mmproj again to produce mmproj-model-f16-4x.gguf.

Quantize to INT4

Once you have the F16 LM GGUF, quantize it with llama-quantize:

Bash

./llama-quantize \
    /path/to/MiniCPM-V-4.7-1B/MiniCPM-V-4_7-1B-F16.gguf \
    /path/to/MiniCPM-V-4.7-1B/MiniCPM-V-4_7-1B-Q4_K_M.gguf \
    Q4_K_M

Other common quant types: Q5_K_M, Q6_K, Q8_0, pick based on your accuracy vs. memory tradeoff. The mmproj file is already small — keep it at F16 and pair it with whichever quantization you chose for the LM.

35B-A3B mixed quantization (recommended)

The MoE expert tensors in 35B-A3B account for about 92% of weights. On top of the default Q4_K_M mixture, reduce just the expert gate / up / down to Q3_K and use imatrix to produce a model around 14.4 GiB with accuracy comparable to Q4_K_M.

tensor-types.txt content:

Plain Text

ffn_(gate|up|down)_exps=Q3_K

Quantize with imatrix:

Bash

./llama-quantize \
    --imatrix /path/to/v47-35a3.imatrix.gguf \
    --tensor-type-file ./tensor-types.txt \
    /path/to/MiniCPM-V-4.7-35B-A3B/MiniCPM-V-4_7-35B-A3B-F16.gguf \
    /path/to/MiniCPM-V-4.7-35B-A3B/MiniCPM-V-4_7-35B-A3B-mix_q3gud_im.gguf \
    Q4_K_M 32

--tensor-type-file is first-match-wins — more specific rules go first.

BNB (int4)

Load the original model, quantize to NF4 with bitsandbytes, run an inference sanity-check, then save the quantized model.

Python

import os
import time
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfig

assert torch.cuda.is_available(), "GPU is required."

model_path = "openbmb/MiniCPM-V-4.7-1B"     # Original model
save_path = "./MiniCPM-V-4.7-1B-int4"       # Save path
image_path = "./assets/airplane.jpeg"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    load_in_8bit=False,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_storage=torch.uint8,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    llm_int8_skip_modules=["lm_head"],
    llm_int8_threshold=6.0,
)

processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForImageTextToText.from_pretrained(
    model_path,
    device_map="auto",
    quantization_config=quantization_config,
)

image = Image.open(image_path).convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "What is in this image?"},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

start = time.time()
with torch.inference_mode():
    out_ids = model.generate(**inputs, max_new_tokens=256)
response = processor.decode(out_ids[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print("Quantized output:", response)
print("Quantized inference time:", time.time() - start)

os.makedirs(save_path, exist_ok=True)
model.save_pretrained(save_path, safe_serialization=True)
processor.save_pretrained(save_path)

The quantized model is saved in save_path, and can be reloaded with AutoModelForImageTextToText.from_pretrained(save_path).

Results (1B):

Item	Value
Precision	4 (NF4, double quant)
quant_method	bitsandbytes
Size	~1.5 GB (bf16 is 3.0 GB)

AWQ

The AutoAWQ workflow for MiniCPM-V is hosted at tc-mb/AutoAWQ (the upstream casper-hansen/AutoAWQ is archived). This fork adds minicpmv4_7 architecture registration.

Use pre-quantized model + vLLM

Python

from PIL import Image
from transformers import AutoProcessor
from vllm import LLM, SamplingParams

MODEL_NAME = "openbmb/MiniCPM-V-4.7-1B-AWQ"
image = Image.open("image.png").convert("RGB")

processor = AutoProcessor.from_pretrained(MODEL_NAME)
llm = LLM(model=MODEL_NAME, max_model_len=8192)

messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "Describe the content of this image"},
    ],
}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

outputs = llm.generate(
    {"prompt": prompt, "multi_modal_data": {"image": image}},
    SamplingParams(temperature=0.0, max_tokens=1024),
)
print(outputs[0].outputs[0].text)

Load AWQ weights with AutoAWQ

Bash

git clone https://github.com/tc-mb/AutoAWQ.git
cd AutoAWQ
pip install -e .

Python

import torch
from PIL import Image
from transformers import AutoProcessor
from awq import AutoAWQForCausalLM

model_path = "openbmb/MiniCPM-V-4.7-1B-AWQ"
image_path = "./assets/airplane.jpeg"

model = AutoAWQForCausalLM.from_quantized(model_path).to("cuda")
processor = AutoProcessor.from_pretrained(model_path)

image = Image.open(image_path).convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "What is in this image?"},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to("cuda")

with torch.inference_mode():
    out_ids = model.generate(**inputs, max_new_tokens=256)
print("Output:", processor.decode(out_ids[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Quantize yourself

Python

import os
import shutil
import torch
from datasets import load_dataset
from transformers import AutoTokenizer
from awq import AutoAWQForCausalLM

model_path = "openbmb/MiniCPM-V-4.7-1B"
quant_path = "./MiniCPM-V-4.7-1B-AWQ"
quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}

model = AutoAWQForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16)
model.model.to("cuda:0")   # Ensure backbone is on GPU
tokenizer = AutoTokenizer.from_pretrained(model_path)

def copy_files_not_in_B(A_path, B_path):
    if not os.path.exists(B_path):
        os.makedirs(B_path)
    files_in_A = {f for f in os.listdir(A_path) if not (".bin" in f or "safetensors" in f)}
    for f in files_in_A - set(os.listdir(B_path)):
        src, dst = os.path.join(A_path, f), os.path.join(B_path, f)
        if os.path.isfile(src):
            shutil.copy2(src, dst)

def load_alpaca():
    data = load_dataset("tatsu-lab/alpaca", split="train")

    def concatenate(x):
        msgs = [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": x["instruction"]},
            {"role": "assistant", "content": x["output"]},
        ]
        return {"text": tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)}

    return [r["text"] for r in data.map(concatenate)][:1024]

calib_data = load_alpaca()
model.quantize(tokenizer, quant_config=quant_config, calib_data=calib_data)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
copy_files_not_in_B(model_path, quant_path)
print(f'Model quantized and saved to "{quant_path}"')

Results (1B):

Item	Value
Quantization method	awq (GEMM)
Precision / group size	4 / 128
Zero point	Yes
Size	~2.3 GB (bf16 is 3.0 GB)

GPTQ

The AutoGPTQ fork for MiniCPM-V is tc-mb/AutoGPTQ. MiniCPM-V 4.7 1B's text backbone is qwen3_5_text, so it reuses the existing Qwen3_5GPTQForCausalLM quantizer.

Use pre-quantized model + vLLM

Python

from PIL import Image
from transformers import AutoProcessor
from vllm import LLM, SamplingParams

MODEL_NAME = "openbmb/MiniCPM-V-4.7-1B-GPTQ"
image = Image.open("image.png").convert("RGB")

processor = AutoProcessor.from_pretrained(MODEL_NAME)
llm = LLM(model=MODEL_NAME, max_model_len=2048)

messages = [{"role": "user", "content": [
    {"type": "text", "text": "(<image>./</image>)\nDescribe the content of this image"},
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

outputs = llm.generate(
    {"prompt": prompt, "multi_modal_data": {"image": image}},
    SamplingParams(temperature=0.0, max_tokens=1024),
)
print(outputs[0].outputs[0].text)

Quantize yourself

Extract the Qwen3.5 text backbone from MiniCPM-V 4.7, quantize it to 4-bit with GPTQ, then reassemble the full model with the original multimodal components.

Bash

git clone https://github.com/tc-mb/AutoGPTQ.git
cd AutoGPTQ
pip install -e .

Python

import os, json, shutil, logging
import torch
from datasets import load_dataset
from transformers import AutoTokenizer, AutoConfig, AutoModelForImageTextToText
from safetensors.torch import load_file as safe_load, save_file as safe_save

MODEL_PATH = "openbmb/MiniCPM-V-4.7-1B"
OUTPUT_PATH = "./MiniCPM-V-4.7-1B-GPTQ"
BITS, GROUP_SIZE = 4, 128
NUM_CALIBRATION_SAMPLES, MAX_SEQ_LEN = 128, 256

def prepare_calibration_data(tokenizer, num_samples, max_length):
    dataset = load_dataset("tatsu-lab/alpaca", split="train")
    dataset = dataset.shuffle(seed=42).select(range(min(num_samples, len(dataset))))
    examples = []
    for sample in dataset:
        text = sample.get("text", "") or sample.get("output", "") or sample.get("instruction", "")
        if not text:
            continue
        tok = tokenizer(text, truncation=True, max_length=max_length, return_tensors="pt")
        examples.append({"input_ids": tok["input_ids"].squeeze(0),
                         "attention_mask": tok["attention_mask"].squeeze(0)})
    return examples

class LLMWrapper(torch.nn.Module):
    def __init__(self, text_model, lm_head, config):
        super().__init__()
        self.model, self.lm_head, self.config = text_model, lm_head, config
        self.seqlen = MAX_SEQ_LEN

    def forward(self, input_ids=None, attention_mask=None, **kwargs):
        hidden = self.model(input_ids=input_ids, attention_mask=attention_mask, **kwargs)[0]
        return self.lm_head(hidden)

tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
examples = prepare_calibration_data(tokenizer, NUM_CALIBRATION_SAMPLES, MAX_SEQ_LEN)

full_model = AutoModelForImageTextToText.from_pretrained(
    MODEL_PATH, dtype=torch.float16, device_map="cpu"
)
text_model, lm_head = full_model.model.language_model, full_model.lm_head
config = full_model.config.text_config
config.model_type = "qwen3_5"

from auto_gptq import BaseQuantizeConfig
from auto_gptq.modeling.qwen3_5 import Qwen3_5GPTQForCausalLM

gptq_model = Qwen3_5GPTQForCausalLM(
    LLMWrapper(text_model, lm_head, config), False,
    BaseQuantizeConfig(bits=BITS, group_size=GROUP_SIZE, desc_act=False, sym=True),
)
gptq_model.quantize(examples, batch_size=1)

# Then reassemble the full state dict: prefix quantized LLM tensors with "model.language_model.",
# merge in non-LLM (vision) tensors, save shards + index + config.json with quantization_config.

Results (1B):

Item	Value
Quantization method	gptq
Precision / group size	4 / 128
desc_act / sym	false / true
Size	~1.9 GB (bf16 is 3.0 GB)

FP8 / INT4 (llm-compressor)

For usable 4-bit on 35B-A3B, you need tools that support fused experts, e.g., llm-compressor (produces compressed-tensors weights).

Published 35B-A3B repos — -FP8, -INT4, -int4, -AWQ, -GPTQ — are all compressed-tensors weights and can be loaded directly with vLLM.

Load pre-quantized weights

Python

from vllm import LLM, SamplingParams
from transformers import AutoProcessor
from PIL import Image

MODEL = "openbmb/MiniCPM-V-4.7-35B-A3B-FP8"   # or -INT4 / -int4 / -AWQ / -GPTQ

processor = AutoProcessor.from_pretrained(MODEL)
llm = LLM(model=MODEL, max_model_len=8192, limit_mm_per_prompt={"image": 5})

image = Image.open("example.jpg").convert("RGB")
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Describe this image"}]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

out = llm.generate({"prompt": prompt, "multi_modal_data": {"image": image}},
                   SamplingParams(stop_token_ids=[248044, 248046], max_tokens=1024))
print(out[0].outputs[0].text)

Quantize yourself

Install llm-compressor (it pins transformers, recommend a separate environment):

Bash

pip install llmcompressor

Recipe — FP8 (W8A8, dynamic):

YAML

# fp8_recipe.yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore:
        - 're:.*lm_head.*'
        - 're:.*vision_tower.*'
        - 're:.*vit_merger.*'
        - 're:.*merger.*'
        - 're:.*linear_attn.*'
        - 're:.*mlp\.gate$'
        - 're:.*shared_expert.*'
        - 're:.*embed_tokens.*'
      scheme: FP8_DYNAMIC
      requires_calibration_data: false

Recipe — INT4 (W4A16, group 128):

YAML

# w4a16_recipe.yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore:
        - 're:.*lm_head.*'
        - 're:.*vision_tower.*'
        - 're:.*vit_merger.*'
        - 're:.*merger.*'
        - 're:.*self_attn.*'
        - 're:.*linear_attn.*'
        - 're:.*mlp\.gate$'
        - 're:.*shared_expert.*'
        - 're:.*embed_tokens.*'
      scheme: W4A16
      requires_calibration_data: false

Run:

Python

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from transformers import AutoModelForImageTextToText, AutoProcessor

MODEL = "openbmb/MiniCPM-V-4.7-35B-A3B"
SAVE = "./MiniCPM-V-4.7-35B-A3B-INT4"

model = AutoModelForImageTextToText.from_pretrained(MODEL, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(MODEL)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["lm_head", "re:.*vision_tower.*", "re:.*vit_merger.*", "re:.*merger.*",
            "re:.*self_attn.*", "re:.*linear_attn.*", "re:.*mlp\\.gate$",
            "re:.*shared_expert.*", "re:.*embed_tokens.*"],
)

oneshot(model=model, recipe=recipe)
model.save_pretrained(SAVE, save_compressed=True)
processor.save_pretrained(SAVE)

The ignore list keeps the vision tower, merger, router / gate, shared expert, linear-attention and embedding at full precision — quantizing them either breaks the canvas M-RoPE vision chain or trades large accuracy loss for small size savings. The fused expert gate_up_proj / down_proj will be quantized, which is the key for MoE models.

Each 35B-A3B quantized repo includes the actual recipe used in its recipe.yaml.