This project provides an optimized default for running MiniCPM5-1B as one active request on an NVIDIA Ampere-or-newer GPU with sufficient VRAM.

RTX 3060 benchmark: 248.52 decode tok/s — 1.503× the matched BF16 vLLM reference.

Benchmarks

Speed

Metric	BF16 reference	FP8 default	Change
Median decode throughput	165.39 tok/s	248.52 tok/s	+50.3%
Median end-to-end throughput	152.42 tok/s	221.42 tok/s	+45.3%
Median TTFT	38.60 ms	35.11 ms	−9.0%

Speed used six prompts, with one warmup and five timed repetitions per prompt. Batch size and concurrency were one.

Quality

Benchmark	BF16 reference	FP8 default	Delta
HumanEval+	95/164	96/164	+1
GSM8K slice	19/50	18/50	−1
MMLU slice	16/50	18/50	+2
C-Eval slice	16/50	18/50	+2
IFEval slice	38/50	33/50	−5
Four-slice total	89/200	87/200	−2

The quality runs were deterministic. FP8 changes outputs: HumanEval+ had eight losses and nine gains relative to BF16; the four-slice suite had 12 losses and 10 gains. Task-level receipts are published under results/ in the repo.

What Was Optimized

The default profile combines:

1. vLLM compiled execution and CUDA graph replay;
2. online per-block FP8 quantization with Marlin linear kernels;
3. Triton attention, the strongest measured decode backend on the RTX 3060;
4. a batch-one scheduler with one maximum active sequence;
5. prefix caching disabled for the single-request target.

The combined profile produced the measured speedup; the repo does not claim a separate percentage for each component.

Quick Start

Measured environment: Linux, RTX 3060 12 GB, vLLM 0.27.1, PyTorch 2.13.0+cu130, model revision 4e9de7a0778dc1c362e983e6858f0e77542cbdca.

Download the official checkpoint directly on the GPU machine:

Bash

hf download openbmb/MiniCPM5-1B \
  --revision 4e9de7a0778dc1c362e983e6858f0e77542cbdca \
  --local-dir /workspace/models/MiniCPM5-1B

Install and launch. FP8 is selected automatically; no profile flag is needed:

Bash

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements-gpu.txt

export MINICPM5_MODEL_PATH=/workspace/models/MiniCPM5-1B
./serve --host 127.0.0.1 --port 8000

Add --dry-run to print the exact vLLM command without starting the server. The launcher checks that the selected NVIDIA GPU has compute capability 8.0 or newer and never downloads weights. It labels the RTX 3060 as measured and other accepted architectures as compatible but unbenchmarked. Available VRAM and the installed vLLM wheel still determine whether startup succeeds.

GPU Compatibility

Hardware	Default profile
NVIDIA Ampere or newer (SM80+)	Architecture-compatible; benchmark separately
RTX 3060 12 GB	Supported and benchmarked here
NVIDIA Turing (SM75)	Not supported by this BF16 profile
AMD, Intel, Apple GPU or CPU	Not supported by this package

vLLM can use Marlin FP8 weight-only kernels on older NVIDIA hardware, but this release also uses BF16, which makes SM80+ the effective floor. Turing would need a separately tested FP16 profile.

What This Release Contains

• the default NVIDIA SM80+ FP8 serving profile and a one-command launcher;
• frozen benchmark specifications and a document-hash manifest;
• full task-level paired comparison receipts;
• a reproducible chart generator and a CPU-safe test suite.

The official OpenBMB weights are unchanged and not redistributed. The BF16 configuration exists only to reproduce the benchmark reference; it is not a second end-user profile. The full protocol is in docs/BENCHMARKS.md in the repo.

Scope and Notes

• The profile is released for NVIDIA SM80+ GPUs. The measured speedup applies directly only to the pinned checkpoint, software, single-request workload and the tested RTX 3060; other GPUs may select different vLLM kernels and must be benchmarked separately.
• The four 50-task domain slices are release screens, not full official leaderboard reproductions.
• This is independent research and not an official OpenBMB release.

Links

• GitHub: vukrosic/minicpm5-nvidia-fp8