MiniCPM-OCR

MiniCPM-V Multimodal Series

MiniCPM-OCR 1.0

Adapt MiniCPM-OCR 1.0 to a specific language, handwriting style or document domain using paired region images and transcriptions. The published example fine-tunes French handwritten line recognition on Teklia/RIMES-2011-line: the input is a text-line image with the fixed prompt Text Recognition:, and the assistant reply is the transcription.

Training uses full-parameter SFT: the vision encoder and language model are trained, the multimodal projector is frozen, and visual token compression is 4×.

Both recipes below are built on tsjyma framework forks pinned to a fixed commit. That commit is a training framework revision, not a MiniCPM-OCR model version: mainline code lacks the 4× visual token compression changes, so the recipe does not reproduce without checking out the pinned commit.

LLaMA-Factory

Environment setup

Source: tsjyma/LlamaFactory, pinned to commit 38fa4bcbac2830c159f2baf9808d21a7fca4e790.

Bash

conda create -n "MiniCPM-OCR-French-LF" python=3.11 -y
conda activate "MiniCPM-OCR-French-LF"

pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1

git clone https://github.com/tsjyma/LlamaFactory.git
cd LlamaFactory
git checkout 38fa4bcbac2830c159f2baf9808d21a7fca4e790
pip install -e .

pip install \
  accelerate==1.13.0 deepspeed==0.18.8 \
  peft==0.18.1 trl==0.24.0 datasets==4.0.0 \
  flash-linear-attention==0.4.2 \
  ninja einops safetensors tokenizers sentencepiece

pip install /path/to/transformers
export DISABLE_VERSION_CHECK=1

Install the Transformers source last; it must be newer than 5.20.2.

Prepare data

Use the official RIMES training, validation and test splits: exclude training samples that contain the ¤ annotation marker, and for validation and test samples remove only the delimiter while keeping the text. Preserve case, accents, punctuation, internal spaces and the original text-line images.

Dataset	Samples
Training	10,138
Validation	1,138
Test	778

Pair each image path with its transcription in JSONL:

JSON

{
  "messages": [
    {
      "role": "user",
      "content": "<image>\nText Recognition:"
    },
    {
      "role": "assistant",
      "content": "J'ai perdu mon emploi depuis 3 mois et je me"
    }
  ],
  "images": [
    "/path/to/rimes_french/images/train/00000.jpg"
  ]
}

JSON

{
  "rimes_train": {
    "file_name": "train.jsonl",
    "formatting": "sharegpt",
    "columns": {
      "messages": "messages",
      "images": "images"
    },
    "tags": {
      "role_tag": "role",
      "content_tag": "content",
      "user_tag": "user",
      "assistant_tag": "assistant"
    }
  },
  "rimes_validation": {
    "file_name": "validation.jsonl",
    "formatting": "sharegpt",
    "columns": {
      "messages": "messages",
      "images": "images"
    },
    "tags": {
      "role_tag": "role",
      "content_tag": "content",
      "user_tag": "user",
      "assistant_tag": "assistant"
    }
  }
}

Launch training

Train for three epochs on one node with 8 × A100 80GB: 80 updates per epoch, 240 in total. Per-device batch size 1 with gradient accumulation 16 gives a global batch size of 128. Evaluate on the full validation set and save a checkpoint after each epoch, then keep the checkpoint with the lowest validation loss.

Training appends thinking\n\n</think>\n\n. User text, image tokens and the empty thinking prefix are masked from loss; the answer and end markers are supervised.
The ds_zero2.json referenced by deepspeed comes from assets/minicpm-ocr/ds_zero2.json in the upstream repository.

Bash

set -euo pipefail
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
export FORCE_TORCHRUN=1 NPROC_PER_NODE=8
export MASTER_ADDR=127.0.0.1 MASTER_PORT=29780
export DOWNSAMPLE_MODE=4x
export DISABLE_VERSION_CHECK=1

DOC_DIR=/path/to/ocr_doc
DATA_DIR=/path/to/rimes_french
OUTPUT_DIR=/path/to/outputs/minicpm_ocr_french_llamafactory
mkdir -p "$OUTPUT_DIR"

cat > "$OUTPUT_DIR/train.yaml" <<YAML
model_name_or_path: /path/to/MiniCPM-OCR-1.0
trust_remote_code: false
flash_attn: sdpa
use_reentrant_gc: false
stage: sft
do_train: true
finetuning_type: full
freeze_vision_tower: false
freeze_multi_modal_projector: true
dataset_dir: ${DATA_DIR}
dataset: rimes_train
template: minicpm_ocr
enable_thinking: false
train_on_prompt: false
cutoff_len: 4096
packing: false
preprocessing_num_workers: 8
dataloader_num_workers: 8
output_dir: ${OUTPUT_DIR}
logging_steps: 1
save_strategy: epoch
save_only_model: false
save_total_limit: 3
report_to: none
per_device_train_batch_size: 1
gradient_accumulation_steps: 16
learning_rate: 5.0e-06
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_steps: 0.05
bf16: true
ddp_find_unused_parameters: true
seed: 42
data_seed: 42
eval_strategy: epoch
weight_decay: 0.1
do_eval: true
deepspeed: ${DOC_DIR}/assets/minicpm-ocr/ds_zero2.json
optim: adamw_torch_fused
adam_beta1: 0.9
adam_beta2: 0.95
adam_epsilon: 1.0e-08
max_grad_norm: 1.0
overwrite_cache: true
eval_dataset: rimes_validation
load_best_model_at_end: true
metric_for_best_model: eval_loss
greater_is_better: false
per_device_eval_batch_size: 1
YAML

llamafactory-cli train "$OUTPUT_DIR/train.yaml"

ms-swift

Environment setup

Source: tsjyma/ms-swift, pinned to commit 903565756360d81fdc16fa14b0340a6e17ace900.

Bash

conda create -n "MiniCPM-OCR-French-Swift" python=3.11 -y
conda activate "MiniCPM-OCR-French-Swift"

pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1

git clone https://github.com/tsjyma/ms-swift.git
cd ms-swift
git checkout 903565756360d81fdc16fa14b0340a6e17ace900
pip install -e .

pip install \
  accelerate==1.13.0 deepspeed==0.18.8 \
  peft==0.18.1 trl==0.24.0 datasets==4.8.4 \
  flash-linear-attention==0.4.2 \
  ninja einops safetensors tokenizers sentencepiece

pip install /path/to/transformers

Install the Transformers source last; it must be newer than 5.20.2.

Prepare data

The RIMES cleaning rules and sample counts are the same as for LLaMA-Factory. The difference is the prompt in the JSONL: the ms-swift template inserts a newline after the image placeholder, so write <image>Text Recognition:.

JSON

{
  "messages": [
    {
      "role": "user",
      "content": "<image>Text Recognition:"
    },
    {
      "role": "assistant",
      "content": "J'ai perdu mon emploi depuis 3 mois et je me"
    }
  ],
  "images": [
    "/path/to/rimes_french/images/train/00000.jpg"
  ]
}

ms-swift does not need dataset_info.json; pass the train.jsonl and validation.jsonl paths directly on the command line.

Launch training

Hyperparameters match LLaMA-Factory: three epochs, 80 steps per epoch, per-device batch size 1, gradient accumulation 16, global batch 128, and the same upstream ds_zero2.json for Deepspeed.

Bash

set -euo pipefail
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
export NPROC_PER_NODE=8
export MASTER_ADDR=127.0.0.1 MASTER_PORT=29781
export DOWNSAMPLE_MODE=4x

DOC_DIR=/path/to/ocr_doc
MODEL_PATH=/path/to/MiniCPM-OCR-1.0
TRAIN_DATA=/path/to/rimes_french/train_swift.jsonl
VAL_DATA=/path/to/rimes_french/validation_swift.jsonl
OUTPUT_DIR=/path/to/outputs/minicpm_ocr_french_swift

swift sft \
  --model "$MODEL_PATH" \
  --model_type minicpm_ocr \
  --template minicpm_ocr \
  --add_non_thinking_prefix true \
  --dataset "$TRAIN_DATA" \
  --split_dataset_ratio 0.0 \
  --deepspeed "$DOC_DIR/assets/minicpm-ocr/ds_zero2.json" \
  --tuner_type full \
  --torch_dtype bfloat16 \
  --freeze_vit false \
  --freeze_aligner true \
  --freeze_llm false \
  --packing false \
  --padding_free false \
  --max_length 4096 \
  --num_train_epochs 3.0 \
  --per_device_train_batch_size 1 \
  --gradient_accumulation_steps 16 \
  --gradient_checkpointing true \
  --vit_gradient_checkpointing true \
  --gradient_checkpointing_kwargs '{"use_reentrant": false}' \
  --learning_rate 5e-06 \
  --lr_scheduler_type cosine \
  --warmup_ratio 0.05 \
  --optim adamw_torch_fused \
  --adam_beta1 0.9 \
  --adam_beta2 0.95 \
  --adam_epsilon 1e-08 \
  --weight_decay 0.1 \
  --max_grad_norm 1.0 \
  --logging_steps 1 \
  --save_total_limit 3 \
  --save_only_model false \
  --load_from_cache_file false \
  --dataset_num_proc 8 \
  --dataloader_num_workers 8 \
  --attn_impl sdpa \
  --loss_scale ignore_empty_think \
  --eval_strategy epoch \
  --output_dir "$OUTPUT_DIR" \
  --report_to none \
  --seed 42 \
  --data_seed 42 \
  --add_version false \
  --use_hf true \
  --dataset_shuffle false \
  --train_dataloader_shuffle true \
  --val_dataset "$VAL_DATA" \
  --save_strategy epoch \
  --load_best_model_at_end true \
  --metric_for_best_model eval_loss \
  --greater_is_better false \
  --per_device_eval_batch_size 1

Evaluation results

Both recipes keep checkpoint-160 from epoch two and evaluate on all 778 official RIMES test lines. The base and fine-tuned models use the same 4× visual token compression, non-thinking mode, greedy decoding, batch size 8 and a maximum of 512 generated tokens per line, with the training prompt Text Recognition:.

CER is the character error rate and WER the whitespace-delimited word error rate. Both use complete model outputs normalised to Unicode NFC with leading and trailing whitespace trimmed, preserving case, accents, punctuation and internal spaces.

Framework	Model	RIMES CER ↓	RIMES WER ↓
LLaMA-Factory	MiniCPM-OCR 1.0	5.67%	15.78%
LLaMA-Factory	Fine-tuned	2.98%	7.66%
ms-swift	MiniCPM-OCR 1.0	5.67%	15.78%
ms-swift	Fine-tuned	3.08%	7.86%

Full tutorials with training curves, per-epoch validation loss and output examples are in the upstream LLaMA-Factory tutorial and ms-swift tutorial.

MiniCPM-OCR Jev

Fine-tune Jev on labelled OCR candidates so quality judgement adapts to your documents. The published recipe attaches a 2-class sequence-classification head and targets 8 × A100 80GB with Python 3.12, ms-swift and DeepSpeed; training and inference use separate environments, and inference needs neither ms-swift nor DeepSpeed.

Environment setup

Training environment

Bash

conda create -n minicpm-ocr-jev python=3.12 -y
conda activate minicpm-ocr-jev

pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
MAX_JOBS=32 NVCC_THREADS=4 pip install --no-build-isolation flash-attn==2.8.3
pip install \
  torch==2.8.0 torchvision==0.23.0 \
  transformers==5.10.4  tokenizers==0.22.2 ms-swift==4.4.0 \
  deepspeed==0.18.9 accelerate==1.13.0 flash-attn==2.8.3 \
  safetensors==0.8.0 pillow==11.3.0 numpy==2.5.1 \
  qwen-vl-utils==0.0.14

Inference / evaluation environment

Bash

pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
pip install \
 torch==2.8.0 torchvision==0.23.0 transformers==5.10.4 \
 tokenizers==0.22.2 safetensors==0.8.0 accelerate==1.13.0 \
 pillow==11.3.0 numpy==2.5.1

transformers>=5.10.4 is a hard requirement: earlier versions lack Qwen3_5ForSequenceClassification and Qwen3VLProcessor. If triton fails to compile during training, point CUDA_HOME at your CUDA installation.

Dataset format

Training data is a quadruple of (original image, OCR text, rendered image, label), with a binary label:

Label	Meaning	Criterion
`1`	OCR text matches the image	Text, formulas and table structure all line up
`0`	Does not match	Any of: missing or wrong characters, broken formula structure, misaligned table rows or columns

The label applies to the whole OCR output with no character-level localisation; the model only answers whether the result is usable. One JSON object per line:

JSON

{
  "messages": [
    {
      "role": "user",
      "content": "<image>\nOCR文本：\n<table><tr><td>Refs</td><td>Accuracy (%)</td></tr></table>\n<image>\n第一张图是原图，第二张图是上述 OCR 文本渲染得到的图片。\n请判断该 OCR 文本及其对应渲染图是否与原图内容一致。\n这是一个二分类任务：一致标为 1，不一致标为 0。"
    }
  ],
  "images": [
    "/path/to/images/gt/case_0025318.png",
    "/path/to/images/label/case_0025318.png"
  ],
  "label": 1
}

images holds exactly two paths, in the order [original, rendered].
The two <image> placeholders in content correspond to those images, with the OCR text in between.
The three closing prompt lines are fixed verbatim; training and inference must use the same prompt.
label is the integer 0 or 1.

Prepare labelled data

Negatives must come from both sources:

Source	How	Why it is needed
Real negatives	Wrong outputs from real OCR models, labelled as mismatched	The error distribution matches production — the class the model actually has to learn
Synthetic negatives	Rule-based perturbation of correct text (change digits, break formula structure, move table cells), then re-render	Covers error types real negatives miss; cheap at volume, with labels correct by construction

Positives are real OCR outputs labelled as matching. Before training, drop rows whose OCR text embeds a base64 image — such text is long enough to crowd out the image token budget:

Bash

python scripts/minicpm-ocr-jev/data/filter_embedded_base64.py \
  --input-dir /path/to/dataset_dir --output-dir /path/to/filtered --splits train

Launch training

Once the environment and data are ready, run this script:

Bash

# run_jev_sft.sh
#!/bin/bash
set -euo pipefail

export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
export NPROC_PER_NODE="${NPROC_PER_NODE:-8}"

# Training-side image token budget; evaluation and inference use the native budget
export IMAGE_MAX_TOKEN_NUM=2048
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

MODEL_PATH="${MODEL_PATH:-/path/to/Qwen3.5-2B}"
TRAIN_DATA="${TRAIN_DATA:-/path/to/filtered/train.jsonl}"
OUTPUT_DIR="${OUTPUT_DIR:-/path/to/outputs/minicpm_ocr_jev}"

swift sft \
  --model "${MODEL_PATH}" \
  --dataset "${TRAIN_DATA}" \
  --task_type seq_cls \
  --num_labels 2 \
  --problem_type single_label_classification \
  --tuner_type full \
  --torch_dtype bfloat16 \
  --freeze_vit true \
  --freeze_aligner true \
  --freeze_llm false \
  --num_train_epochs 3 \
  --per_device_train_batch_size 1 \
  --gradient_accumulation_steps 1 \
  --learning_rate 1e-5 \
  --lr_scheduler_type cosine \
  --warmup_ratio 0.05 \
  --weight_decay 0.1 \
  --max_grad_norm 1.0 \
  --max_length 12000 \
  --packing true \
  --packing_length 12000 \
  --packing_num_proc 8 \
  --deepspeed zero3 \
  --gradient_checkpointing true \
  --attn_impl flash_attn \
  --logging_steps 10 \
  --eval_strategy no \
  --save_strategy steps \
  --save_steps 500 \
  --save_total_limit 40 \
  --save_only_model true \
  --dataset_num_proc 48 \
  --dataloader_num_workers 4 \
  --ddp_timeout 10800 \
  --seed 42 \
  --output_dir "${OUTPUT_DIR}" \
  --report_to tensorboard

Key arguments:

--task_type seq_cls --num_labels 2 --problem_type single_label_classification turn the base model from a generator into a scorer, attaching the 2-way head on the last token.
--packing true --max_length 12000: two images plus OCR text make long sequences, and throughput drops sharply without packing.
IMAGE_MAX_TOKEN_NUM=2048 applies to training only. Evaluation and inference use the checkpoint preprocessor's native budget; this asymmetry is intentional, do not harmonise it.
To change GPU count, set CUDA_VISIBLE_DEVICES and NPROC_PER_NODE; global batch = GPUs × per-device batch × gradient accumulation.

Evaluation

Evaluation runs on JevOCRBench, a block-level benchmark for OCR verifiers. Each sample is a block image, one OCR candidate and a rendering of that candidate; the ground truth is a human judgement of correct or incorrect, and the metric is AUROC.

Bash

huggingface-cli download openbmb/JevOCRBench --repo-type dataset \
  --local-dir /path/to/JevOCRBench

Score and compute AUROC on a single GPU:

Bash

# 1. Score: one prob_1 per sample
python scripts/minicpm-ocr-jev/eval/score.py \
  --dataset /path/to/JevOCRBench/data/jevocrbench.jsonl \
  --checkpoint /path/to/weights \
  --out /path/to/predictions.jsonl

# 2. Compute AUROC: overall and per category
python scripts/minicpm-ocr-jev/eval/evaluate.py \
  --predictions /path/to/predictions.jsonl

Leaderboard results for released models are in the upstream README, and the full recipe is in finetune_jev_en.md.

Best practices

Split by page, never by sample — regions from one page are highly correlated.
Keep both negative sources. Synthetic-only data trains a model that fails on real OCR errors; real-only data leaves error types uncovered.
Freeze the renderer. A new renderer version is a new input distribution, so old numbers must be re-measured.
Keep the prompt byte-identical to training at evaluation and inference, and check the last token id.

MiniCPM-OCR 1.0​

LLaMA-Factory​

Environment setup​

Prepare data​

Launch training​

ms-swift​

Environment setup​

Prepare data​

Launch training​

Evaluation results​

MiniCPM-OCR Jev​

Environment setup​

Dataset format​

Prepare labelled data​

Launch training​

Evaluation​

Best practices​

MiniCPM-OCR 1.0

LLaMA-Factory

Environment setup

Prepare data

Launch training

ms-swift

Environment setup

Prepare data

Launch training

Evaluation results

MiniCPM-OCR Jev

Environment setup

Dataset format

Prepare labelled data

Launch training

Evaluation

Best practices