MiniCPM-o 4.5 goes open source: eyes, ears, and voice together, from Q&A to instant free-flowing conversation
Today, we are open-sourcing our new flagship omni-modal model, MiniCPM-o 4.5. As a natively full-duplex omni-modal large model, MiniCPM-o 4.5 takes human-AI interaction to a new level. With omni-modal capabilities that let it watch, listen, and speak up on its own all at once, it frees AI from the rigid, walkie-talkie-style turn-based interaction and becomes the industry’s first large model built for instant, free-flowing conversation — perception never stops, dialogue never feels scripted, and reminders come without asking. At just 9B parameters, it packs state-of-the-art capabilities such as high-frame-rate visual understanding and end-to-end speech conversation into one model, opening a new era of human-like perception and communication.
MiniCPM-o 4.5 is now open-source on GitHub, Hugging Face, and other platforms. Together with llama.cpp-omni, ModelBest’s self-developed inference framework — the industry’s first open-source, high-efficiency on-device inference framework for streaming omni-modal models — it makes deployment simpler, more stable, and more efficient. In addition, leveraging the cross-platform capabilities of the unified system software stack FlagOS, MiniCPM-o 4.5 achieves end-to-end inference performance gains on six chips, including Iluvatar CoreX, Huawei Ascend, T-Head, Hygon, and MetaX. We also offer a no-deployment online demo for developers, and we invite you to explore new capabilities and application scenarios with MiniCPM-o 4.5.
➤ MiniCPM-o 4.5 highlights at a glance
Setting a new ceiling for on-device omni-modal capabilities: leading performance in omni-modal understanding, visual understanding, document parsing, speech understanding and generation, and voice cloning, with the best inference efficiency and the lowest inference cost.
All-modality, always-on perception: inputs and outputs across vision, audio, and text no longer block each other. Even while generating output, the model is still watching and listening, so no information is lost.
From turn-based interaction to instant, free-flowing conversation: by staying aware of its surroundings at all times, the model can reply in real time, at the right moment and with the right content.
Natural, expressive speech: significant gains in timbre, human-likeness, and expressiveness for omni-modal end-to-end speech generation. Voice cloning supports fully custom timbres, and long-form speech synthesis is more stable.
First-of-its-kind native full-duplex technology: an end-to-end omni-modal architecture combined with a full-duplex multimodal real-time streaming mechanism, a proactive interaction mechanism, and configurable speech modeling gives MiniCPM-o 4.5 interaction that feels as natural as talking to a person.
➤ Model links
GitHub:
🔗 https://github.com/OpenBMB/MiniCPM-o
HuggingFace:
🔗 https://huggingface.co/openbmb/MiniCPM-o-4_5
ModelScope:
🔗https://www.modelscope.cn/models/OpenBMB/MiniCPM-o-4_5
Modelers:
🔗https://modelers.cn/models/OpenBMB/MiniCPM-o-4_5
Gitcode:
🔗 https://ai.gitcode.com/OpenBMB/MiniCPM-o-4_5
Demo link — full-duplex omni-modal mode:
🔗 https://huggingface.co/spaces/openbmb/minicpm-omni
Demo link — image-text chat mode:
🔗 http://211.93.21.133:18121/
The strongest on-device omni-modal model: leading in seeing, listening, and speaking, and raising the bar again
Following the “high density” approach that defines ModelBest’s MiniCPM series, MiniCPM-o 4.5 delivers SOTA-level omni-modal performance across omni-modal understanding, visual understanding, document parsing, speech understanding and generation, and voice cloning — with just 9B parameters!
Swipe left to see more
MiniCPM-o 4.5 not only raises the bar on capability density; it also pursues extreme efficiency. Lower VRAM usage and faster response times ensure the best inference efficiency and the lowest inference cost while still delivering SOTA-level omni-modal performance.
Full-duplex: eyes, ears, and voice working together, leaving walkie-talkie-style interaction behind
Last year, MiniCPM-o 2.6 gave on-device large models the ability to keep watching, listen in real time, and speak naturally. This year we bring another revolutionary upgrade: omni-modal models that can watch, listen, and speak up on their own at the same time, with full-duplex and proactive interaction.
Conventional simplex models work like a walkie-talkie: while answering, they cannot take in outside information and can only “finish speaking, then look; finish speaking, then listen.” It is like covering your eyes and ears while talking — the interaction never feels natural, and the most critical information is easily lost.
“Eyes on everything, ears on every side”
MiniCPM-o 4.5 can watch, listen, and speak up on its own, staying perceptive at all times no matter the situation. Combined with large-model understanding of HD video and sharp, responsive sound recognition, it misses no critical detail and is not thrown off by noisy information.
This article includes video content. Please view it in the original WeChat post.
“Adapting on the fly, acting at the right moment”
Most multimodal models still rely on external tools such as VAD (Voice Activity Detection) and engineering workarounds to control when they speak. As a natively full-duplex model, MiniCPM-o 4.5 has another advantage: while staying perceptive, it reacts to changes in its environment in real time and replies at the right moment with the right content, so its perception and delivery of information are never a beat behind.
Natural speech: a fully upgraded voice that sounds expressive and remarkably human
Human-like perception and interaction require more than continuous watching and listening; they also require speech that is more stable, more natural, and more expressive.
Through new model design and data methods, MiniCPM-o 4.5 improves timbre, human-likeness, and expressiveness across the board. During audio output it automatically picks the most fitting tone and timbre, producing a voice that sounds like a real person.
Notably, MiniCPM-o 4.5 also largely solves the inconsistent timbre, unnatural tone, and unstable output that plague long-form speech synthesis; even very long generations stay stable.
For voice cloning, MiniCPM-o 4.5 offers more voice options: a few seconds of audio are enough to clone and customize a new timbre, which can then be used for role-play voice conversations.
Model architecture
MiniCPM-o 4.5 uses an end-to-end omni-modal architecture and innovatively adds a full-duplex multimodal real-time streaming mechanism, a proactive interaction mechanism, and configurable speech modeling, giving it interaction that feels as natural as talking to a person.
End-to-end omni-modal architecture. The encoders/decoders for each modality are tightly connected to the LLM end-to-end through dense features. This design enables better information flow and control, and helps fully exploit rich multimodal knowledge during training.
Full-duplex multimodal real-time streaming.
MiniCPM-o 4.5 turns offline modality encoders/decoders into online full-duplex versions that support streaming input/output. The speech decoder uses interleaved text and speech token modeling, enabling full-duplex speech generation (that is, synchronized with new input in real time) while also improving the stability of long speech generation (for example, longer than one minute).
Time-division multiplexing: MiniCPM-o 4.5 synchronizes all input and output streams on a millisecond-level timeline and uses time-division multiplexing for unified modeling in the language model backbone. This mechanism divides parallel omni-modal streams into sequential information groups within tiny periodic time slices, enabling efficient omni-modal streaming.
Proactive interaction mechanism. The language model module continuously monitors the incoming video and audio streams and automatically decides at 1 Hz whether to speak. This high-frequency decision-making, combined with full duplex, is key to proactive interaction such as unprompted reminders and comments.
Configurable speech modeling. MiniCPM-o 4.5 continues the multimodal system prompt design of MiniCPM-o 2.6, with both a text system prompt and an audio system prompt (used to specify timbre). This allows the model to perform voice cloning and role-play at inference time with a simple reference audio clip.
How to use the model
MiniCPM-o 4.5 can be deployed and used in many ways:
(1) Efficient CPU inference on local devices via llama.cpp and Ollama;
(2) 16 int4 and GGUF quantized models in different sizes;
(3) High-throughput, memory-efficient inference via vLLM and SGLang;
(4) Multi-chip backend support for MiniCPM-o 4.5 through the unified system software stack FlagOS. Users can get a multi-chip version of MiniCPM-o 4.5 out of the box from the public FlagOS image, or launch it with vLLM-plugin-FL;
(5) Fine-tuning for new domains and tasks with LLaMA-Factory;
(6) An online web demo. We have also released the high-performance llama.cpp-omni inference framework and a companion WebRTC demo, bringing the full-duplex multimodal real-time streaming experience to PCs and other local devices.
➤ Try it online
🔗https://huggingface.co/spaces/openbmb/minicpm-omni
Livestream and in-person meetup invitation
Dear developers,
The OpenBMB open-source community sincerely invites you to a technical meetup on “ModelBest MiniCPM-o 4.5,” streamed live online. We are honored to welcome Yao Yuan, Assistant Professor at the Institute for AI at Tsinghua University and Chief Scientist of Multimodal at ModelBest, to discuss the frontiers of multimodal technology and its practical applications. We look forward to meeting everyone interested in this field offline to exchange ideas! ❤️
📅 Time: February 7 (Saturday), 19:00
👉 Watch online: tune in to the official OpenBMB WeChat Channels / Bilibili account and join the livestream on time
👉 Join in person: a small number of in-person seats are now open! The venue is near Tsinghua University. Scan the QR code below to fill out the registration form — seats are limited and first come, first served.
*Successful registration will be confirmed by SMS, so please make sure your contact details are correct!
We look forward to exploring the latest advances in multimodal technology with you! ❤️