MiniCPM-o 4.5 Released: Toward Real-Time Full-Duplex Omni-Modal Interaction

Introduction: A Paradigm Shift in Multimodal Interaction

The ModelBest team recently published a new paper on arXiv, officially unveiling the MiniCPM-o 4.5 model. The model addresses a long-standing core challenge in the field of multimodal large language models (MLLMs) — how to achieve truly Real-Time Full-Duplex Omni-Modal Interaction. This work marks a significant breakthrough in AI's transition from static offline data processing to real-time streaming interaction.

The Core Problem: The Interaction Paradigm Itself Becomes the Bottleneck

In recent years, progress in multimodal large language models has been remarkable. From GPT-4o to Gemini, models have steadily improved in modality coverage and response latency. However, the paper points out that the key bottleneck preventing AI from reaching human-level multimodal interaction is no longer just about modality coverage or latency — it is the interaction paradigm itself.

Specifically, existing models face two core challenges:

• Alternating separation of perception and response: Traditional multimodal models adopt a half-duplex "listen first, then speak" mode, where perception and generation are strictly divided into alternating phases. This means the model cannot simultaneously receive new input while generating a response, and cannot interrupt, supplement, or adjust on the fly as in human conversation.

• Lack of true real-time streaming capability: Even when some models support streaming input, their internal processing still operates in a "batch and process" mode, unable to achieve frame-by-frame real-time understanding and response to continuous input signals.

These limitations result in a significant gap between current multimodal AI interaction experiences and real human communication.

Technical Approach: Architectural Innovations of MiniCPM-o 4.5

MiniCPM-o 4.5 proposes a systematic solution to the above problems. Its core design philosophy is to merge perception and generation into a unified, parallelizable pipeline, thereby enabling full-duplex interaction.

Full-Duplex Architecture Design

Unlike the traditional "turn-taking" dialogue mode, MiniCPM-o 4.5 adopts a full-duplex architecture that allows the model to continuously receive and process new input signals while outputting responses. This design endows the model with the following capabilities:

• Real-time interruption and response adjustment: Users can insert new information at any point during the model's response, and the model can immediately perceive and adjust its output.
• Uninterrupted continuous perception: Perceptual channels such as vision and hearing remain active at all times and are not paused while the model is generating a response.
• Natural conversational rhythm: The interaction more closely approximates the natural rhythm of face-to-face human communication, including real-time capture of filler words, pauses, and emotional changes.

Omni-Modal Fusion Capability

The "Omni-Modal" in the model's name reflects its unified processing capability across multiple modalities. MiniCPM-o 4.5 not only supports common modalities such as text, image, and speech, but also strives to unify the understanding and generation of these modalities within a coherent framework, avoiding information fragmentation between different modalities.

Feasibility of On-Device Deployment

The MiniCPM series has always been known for being "small yet powerful," and MiniCPM-o 4.5 continues this tradition. While maintaining robust multimodal capabilities, it keeps the parameter scale under control, giving it the potential for deployment on edge devices. This is of great significance for real-time interaction scenarios that demand low latency and strong privacy protection.

In-Depth Analysis: Why Full-Duplex Interaction Matters

From Tool to Companion

The dominant AI interaction mode today is essentially a "command-response" tool-usage paradigm. Users issue commands and wait for AI to finish processing before returning results. The realization of full-duplex interaction means AI can perceive a counterpart's feedback in real time during a conversation and dynamically adjust its expression, much like a human companion — this represents a fundamental leap from "AI tool" to "AI companion."

Technical Difficulties and Challenges

Achieving full-duplex omni-modal interaction poses numerous technical challenges:

1. Real-time scheduling of computational resources: Simultaneously performing perception and generation requires fine-grained resource allocation strategies that ensure real-time processing of input signals while maintaining output fluency.
2. Dynamic context updates: Continuously incorporating new perceptual information during generation requires the model to have a flexible context management mechanism.
3. Temporal alignment of multimodal signals: Different modality signals have varying sampling rates and processing latencies; achieving precise alignment in the temporal dimension is both an engineering and algorithmic challenge.
4. Matching training data with the paradigm: The full-duplex interaction mode lacks large-scale, high-quality training data, making the construction of effective training pipelines a critical issue.

Competitive Landscape

In the omni-modal interaction arena, OpenAI's GPT-4o was the first to demonstrate impressive real-time voice interaction capabilities, and Google's Gemini series continues to push forward in multimodal real-time understanding. The unique value of MiniCPM-o 4.5 lies in its attempt to achieve comparable or even superior interaction experiences at a much smaller model scale, which is of strategic importance for on-device deployment and mass adoption.

Domestically, Alibaba's Qwen series and Zhipu AI's GLM series are also actively expanding in the multimodal space, but in the specific niche of full-duplex real-time interaction, MiniCPM-o 4.5 demonstrates a clear technical roadmap and first-mover advantage.

Application Outlook

The maturation of full-duplex omni-modal interaction technology will unlock a range of entirely new application scenarios:

• Intelligent companionship and education: AI tutors can observe students' facial expressions and tone of voice in real time while they answer questions, assess comprehension levels, and instantly adjust teaching strategies.
• Remote collaboration and meetings: AI assistants can continuously understand multi-party speech, shared screen content, and body language during meetings, providing real-time assistance.
• Accessible interaction: More natural real-time multimodal information translation services for people with hearing or visual impairments.
• Smart cockpits and robotics: In automotive and embodied intelligence scenarios, full-duplex interaction capability is the foundation for safe and natural human-machine collaboration.

Conclusion and Outlook

The release of MiniCPM-o 4.5 represents an important turning point for multimodal large models — shifting from "capability stacking" to "interaction paradigm innovation." The paper clearly states that the core competitiveness of future multimodal AI lies not only in "what it can understand" but also in "how it interacts with people." Full-duplex, real-time, omni-modal — these three keywords outline the basic contours of next-generation AI interaction.

Although there is still a gap between the paper and product deployment, the technical direction explored by MiniCPM-o 4.5 undoubtedly carries far-reaching implications. As computational efficiency continues to improve and training methods are further optimized, there is every reason to expect that the era of truly natural and fluid human-machine multimodal interaction is accelerating toward us.