MiniCPM-o 2.6 is an 8-billion-parameter omni-modal model supporting real-time speech dialogue and multimodal live streaming.
Technical highlights
End-to-end omnimodal architecture：Encoders/decoders of different modalities are connected and trained in an end-to-end manner to fully exploit the rich omnimodal knowledge.
Omnimodal streaming mechanism：We changed the offline modal encoder/decoder to an online version to handle streaming input/output. We design a time-division multiplexing (TDM) mechanism for omnimodal stream processing in the LLM backbone. It splits parallel omnimodal streams into sequential information in small time slices.
Configurable speech modeling design：We designed a multimodal system prompt that includes traditional text system prompts, as well as a new audio system prompt to determine the assistant's voice. This enables flexible sound configuration at inference time and also facilitates end-to-end sound cloning and description-based sound creation.