MiniCPM-o 4.5 technical share: how can multimodal large models be more human? | AGI Open Source Salon 001
On the evening of February 7, 2026, the OpenBMB open-source community and AGI BAR co-hosted a technical sharing session titled "MiniCPM-o 4.5: How Can Multimodal Large Models Be More Human?" At the session, Yao Yuan, Chief Scientist of Multimodal at ModelBest and Assistant Professor at Tsinghua University, started from the newly open-sourced MiniCPM-o 4.5 full-duplex omni-modal large model and gave an in-depth talk on its technical principles, capability breakthroughs, and future applications.
Yao Yuan is an Assistant Professor at the College of Artificial Intelligence, Tsinghua University, and Chief Scientist of Multimodal at ModelBest.
His main research focuses on multimodal large models and natural language processing. He has published more than 30 papers at top international conferences and journals such as Nature Communications, CVPR, ACL, and ICLR, with multiple selections as conference highlights or journal recommended articles. His Google Scholar citations exceed 8,500, his GitHub open-source projects have more than 36,000 stars, and his open-source models have been downloaded more than 16 million times. He led the construction of the efficient multimodal large model series MiniCPM-V and MiniCPM-o; the series was selected for the Hugging Face 2024 most popular and most downloaded open-source models list and the 10 major scientific and technological achievements of the Zhongguancun Forum Annual Meeting. He has received honors including the CAAI Outstanding Doctoral Dissertation Award, the Second Prize of the Ministry of Education's Natural Science Research Achievement Award, the Intel China Academic Achievement Award, and the World Artificial Intelligence Conference Yunfan Award.
The livestream drew 35,000 developers and researchers. Yao Yuan systematically explained three major innovations in MiniCPM-o 4.5's human-like "watch, listen, and speak proactively" interaction, and publicly disclosed for the first time the key implementation paths behind its full-duplex architecture, autonomous interaction mechanism, and human-level speech generation. In the Q&A session that followed, onsite and online attendees from robotics, VR, education, AIoT, and other fields asked questions enthusiastically, leading to in-depth discussion on on-device deployment, training strategies, multimodal understanding, and other topics.
In this talk, Yao Yuan systematically introduced three major innovations in MiniCPM-o 4.5's human-like "watch, listen, and speak proactively" interaction, and publicly disclosed for the first time the key implementation paths behind its full-duplex architecture, autonomous interaction mechanism, and human-level speech generation. MiniCPM-o 4.5 was jointly built by teams from Tsinghua University x ModelBest x OpenBMB.
The event used a hybrid format of "online livestream + offline exchange," and the livestream drew 35,000 developers and researchers. In the interactive Q&A after the talk, onsite and online attendees from robotics, VR, education, AIoT, and other fields also asked questions enthusiastically, with in-depth discussion on on-device deployment, training strategies, multimodal understanding, and other topics.
Highlights from the talk
To open the talk, Yao Yuan reviewed the development path of the MiniCPM efficient multimodal model series.
From open-sourcing an on-device multimodal model in early 2024 to subsequent breakthroughs in OCR, multi-image and video understanding, and vision-speech-streaming capabilities, the team has consistently pursued leading foundational capabilities at the same parameter scale, closely combining frontier lab exploration with productization needs.
He noted that most multimodal models still follow the language model's "offline, turn-based dialogue" interaction paradigm — processing static data and engaging in one-question-one-answer exchanges. Humans, however, live in a continuous and parallel multimodal information stream, able to perceive in real time, react with low latency, and freely decide when to speak.
To close this fundamental gap, MiniCPM-o 4.5's core exploration centers on "how to make multimodal large models more human," focusing on three key human interaction characteristics: non-blocking omni-modal I/O streams, freedom of autonomous interaction over multimodal streams, and natural, low-latency, human-feeling voice conversation.
To achieve these goals, MiniCPM-o 4.5 made a fundamental architectural innovation.
The model uses an end-to-end design of an "LLM backbone + lightweight modality encoders/decoders." While retaining strong text capabilities, it achieves a high capability ceiling through tight hidden-layer connections. On this basis, the team pioneered a "full-duplex omni-modal" modeling mechanism. By aligning and slicing input and output video, audio, text, and speech streams on the timeline with high precision (for example, one processing unit per second), the model achieves parallel processing similar to a human's ability to "watch, listen, and speak at the same time," with input and output not blocking each other and strategies adjusted in real time based on environmental feedback.
To address traditional models' passive conversation timing and reliance on external VAD (voice activity detection) tools, MiniCPM-o 4.5 introduces a high-frequency autonomous interaction semantic judgment mechanism. The model can decide once per second whether it should speak, thereby natively supporting human-like interaction behaviors such as instant replies, interruptions, proactive responses, and interjections, truly weakening the rigid turn-based dialogue experience.
On the speech interaction side, thanks to token-level end-to-end connections, the LLM backbone can handle complex instruction understanding and prosody planning, guiding a lightweight TTS decoder to generate natural, expressive speech — even character-level prosody control. Meanwhile, the full-duplex streaming speech generation mechanism and a "look-ahead" technique for upcoming context ensure low latency while maintaining the coherence and stability of long speech. The model also innovatively supports "multimodal system prompts," letting users easily achieve high-quality end-to-end voice cloning and role-play by combining text with a reference audio clip.
Beyond frontier interaction capabilities, MiniCPM-o 4.5 remains at the leading level among same-size models in foundational multimodal abilities such as image-text understanding, document parsing, and video Q&A. To make it easier for developers, the model has been adapted to Llama.cpp, ollama, and several other mainstream inference frameworks. The team also built llama.cpp-omni, the first efficient on-device streaming omni-modal inference framework, and open-sourced a lightweight local demo system so users can experience full-duplex omni-modal interaction on a personal computer.
Q&A highlights
Q1: What challenges arise from integrating full-duplex and omni-modal capabilities all in one?
Yao Yuan: There are two main challenges.
One is the balancing act of fusing multimodal capabilities.
It is commonly observed in the industry that when trying to unify vision, speech, text, and other modality capabilities, along with complex tasks such as instruction following (instruct) and deep thinking (thinking), into one model, performance "drops" easily — different capabilities interfere with each other. This requires the team to have a deeper understanding and precise grasp of model architecture design, data ratios, and the characteristics of different modalities during learning. After repeated iteration and optimization, we finally achieved a good balance and coordinated improvement of multiple complex capabilities under a unified architecture.
The second is blazing a trail through the "uncharted territory" of systemic innovation.
Full-duplex omni-modal is a frontier direction with no mature precedent in the open-source community. That means everything from the most fundamental model architecture, runtime mechanisms, and infra to training data construction methods and training strategies, and further up to the evaluation system, efficient inference framework, and even the interactive demo system — the entire stack requires coordinated design and innovation starting from zero. This is a typical exploration of uncharted territory, and the hardest part is the lack of a reference path: the whole team must advance multiple pieces in parallel amid uncertainty and ensure they ultimately couple into a usable whole.
Q2: Where are the unique application prospects for these more human-like models?
Yao Yuan: Between an "AI that mechanically executes commands" and an "AI that is completely human-like," there is a broad capability spectrum, and that is exactly where full-duplex omni-modal models can unlock unique value. The core is the ability to "watch and listen while interacting proactively at the right moment based on an understanding of the environment." For example:
Companion partner: While you cook, it can see the water boiling and proactively remind you; in an elevator, it can continuously read floor changes and naturally tell you when you reach your target floor.
Accessibility assistance: For visually impaired users, it is no longer "you ask, I answer" but continuous environmental observation and proactive announcements of key events (such as a green light turning on or a cup about to be full).
Smart cockpit: While driving, it continuously observes the road, proactively points out "an available parking space on the left," and guides parking.
Embodied AI and robots: As the robot's "brain," it continuously perceives the dynamic environment and autonomously decides when to interact and act.
What these scenarios share is that the need is not one-off Q&A; the model must integrate into the dynamic flow of life as a "silent observer" and "timely reminder" — exactly what traditional turn-based dialogue models cannot do.
Q3: Why emphasize that human-like AI models must run on on-device hardware such as phones?
Yao Yuan: This is not a simple technical choice but an inevitable path determined by the essential requirements of "human-like companion AI."
A truly human-like, always-on companion AI will see and hear the user's most private life information. Uploading such continuous and private multimodal data streams to the cloud carries an unacceptable risk of privacy leakage. Keeping data on the device is the best protection for user privacy.
Human-like interaction requires extremely low latency and high stability. Network fluctuations and signal dead zones (such as tunnels or the wilderness) would severely undermine the "real-time perception and response" experience. Only on-device compute can provide deterministic low latency and high availability.
Moreover, if hundreds of millions of users relied on the cloud to process uninterrupted 24/7 full-duplex video and audio streams, the compute and bandwidth required would be astronomical — unsustainable in both cost and energy. Distributing compute across every endpoint and using local resources to close the loop is the most viable architecture for building inclusive, sustainable large-scale services.
Therefore, the exploration of making large models "more human" and the practice of running powerful intelligence "on device" are two sides of the same coin, both pointing toward a future that is smarter, more privacy-secure, and more inclusive.
More technical details on MiniCPM-o 4.5
Q: Does the model consume context quickly? How long can it interact continuously?
A: We currently treat full-duplex as an experimental feature and recommend using it within a context window of about 1 minute for the best results. In testing, continuous interaction within 3 minutes also performed well. Longer interactions involve storing and evicting large-scale KV Cache, a frontier problem for the industry that will require joint breakthroughs in long-context model techniques and inference frameworks.
Q: Does the model support further fine-tuning?
A: Fine-tuning a single capability (such as image-text understanding or voice conversation) is feasible, and we provide scripts. But high-quality fine-tuning of omni-modal, full-duplex capabilities as a whole currently faces challenges in high-quality data construction and infra, and will require joint effort from the community.
Q: Is the full-duplex approach much more expensive than traditional half-duplex or engineering pipeline approaches?
A: An engineering pipeline approach (such as taking a photo and then recognizing it) cannot achieve true real-time memory and companion-style interaction. A half-duplex model cannot "listen, speak, and adjust at the same time." What full-duplex brings is a fundamental expansion of capability forms that can solve many more challenging problems. If a pipeline can meet a specific need, it is indeed a stable, low-cost choice. But when a scenario requires the model to truly understand and blend into a dynamic environment "like a human," full-duplex is the necessary technical path.
Q: For devices with limited on-device compute such as robots, how long until omni-modal models run successfully?
A: It depends on coordinated progress in algorithms, chips, and scenarios. On algorithms, the "density" of model performance is rising quickly, and the compute needed for equivalent capability is falling. On hardware, on-device chip compute is growing along Moore's Law, and chips designed specifically for Transformers are coming soon. Optimistically, within the next 1 to 2 years, we can expect to see practical on-device omni-modal models on robots and other devices.
Q: How can long-term multimodal memory be achieved, for example reviewing surveillance video from a day ago?
A: Having coherent long-term memory like a human is a core challenge. One straightforward approach is RAG (retrieval-augmented generation): retrieve historical video clips and feed them into the model. But this is not the most elegant or efficient way. How to let a model autonomously form, store, and recall highly abstract multimodal memories like a human is a research direction worth long-term exploration.
Q: What is the theoretical basis for small models achieving high performance through "intelligence density"? Is there a limit?
A: It comes from the fact that different models convert data and compute into intelligence at different efficiencies. The human brain is not the largest, yet it is the smartest. The key lies in coordinated optimization of model architecture, training methods, and data quality, not just parameter count. For example, to build OCR capability, adding more OCR data early on did not help much, but switching to high-resolution visual encoding produced a leap in performance. So the improvement of intelligence density is far from its limit. The so-called limit is more of an engineering question of "achieving a sufficiently good experience under specific hardware constraints," and those constraints keep loosening as hardware evolves.
➤ Model links
GitHub:
🔗 https://github.com/OpenBMB/MiniCPM-o
Hugging Face:
🔗 https://huggingface.co/openbmb/MiniCPM-o-4_5
ModelScope:
🔗 https://www.modelscope.cn/models/OpenBMB/MiniCPM-o-4_5
Modelers:
🔗 https://modelers.cn/models/OpenBMB/MiniCPM-o-4_5
GitCode:
🔗 https://ai.gitcode.com/OpenBMB/MiniCPM-o-4_5
The OpenBMB community has always believed that the vitality of frontier technology lies in open collision and collaboration. To that end, we are committed to continuing and increasing our open-source investment, providing developers with full-stack support from models to deployment.
Inference deployment
01 General use
Llama.cpp, ollama, vLLM, SGLang, FlagOS, and other frameworks fully supported
02 Using omni-modal capabilities
>> Inference deployment framework
llama.cpp-omni, efficient on-device inference
>> Demo systems
Web: a WebRTC-based web demo interaction system
Local: a lightweight demo interaction system is now open source, with two deployment options ⬇️
Option 1: 🐳 Deploy with Docker — skips environment installation. See the document below; supports macOS / Linux / Windows. 🔗 https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/demo/web_demo/WebRTC_Demo/README_zh.md
Option 2: 🚀 Use oneclick.sh for one-click installation and deployment — a native deployment option that is friendlier to memory and easier to customize; supports macOS / Linux. 🔗 https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/demo/web_demo/WebRTC_Demo/oneclick.md
⚙️ Hardware requirements
Half-duplex voice 🖥️ Mac: at least 16 GB RAM; M4 chip + 32 GB or more RAM recommended. 🎮 CUDA (Linux / Windows): 12 GB+ VRAM recommended.
Full-duplex Omni 🖥️ Mac: at least 32 GB RAM; M4 Max + 64 GB or more RAM recommended. 🎮 CUDA (Linux / Windows): 16 GB+ VRAM recommended.
We are also continuously recruiting like-minded partners to push the technical frontier together. Welcome to join us 👋 ❤️!