MoSS: Introducing Modular Multi-Sensory Fusion for VLA Models

From 'Seeing' to 'Feeling': A Perceptual Upgrade for Embodied Intelligence

Human interaction with the real world has never relied solely on vision — we perceive an object's texture and hardness through touch, gauge gripping force through force sensing, and coordinate body movements through proprioception. Yet today's mainstream Vision-Language-Action models (VLAs) still depend heavily on visual information when endowing robots with intelligence, leaving a critical gap in their physical "somatic" understanding of the world.

A new paper recently published on arXiv (arXiv:2604.23272v1) introduces an innovative framework called MoSS (Modular Sensory Stream), designed to seamlessly integrate multiple physical sensory signals into VLA models in a modular fashion, opening entirely new perceptual dimensions for embodied intelligence.

The MoSS Framework: A Modular Design to Solve Multi-Sensory Fusion Challenges

Limitations of Existing Approaches

In recent years, several studies have attempted to incorporate physical sensory signals into VLA models, but these approaches typically focus on only a single type of physical signal — for example, connecting only a tactile sensor or fusing only torque data. This "single-sense" design has obvious shortcomings: real-world interaction is inherently heterogeneous and complementary, and a single signal cannot fully characterize complex physical environments. For instance, when a robot manipulates fragile objects, it needs tactile feedback to assess contact state, force sensing to control applied pressure, and possibly auditory perception to detect anomalies.

The Core Philosophy of MoSS

The design philosophy of MoSS can be summed up in one word: modularity. Rather than treating sensory fusion as a monolithic problem, the framework designs an independent sensory stream module for each physical signal, enabling:

• Flexible Integration: Different types of physical sensors (tactile, force, proprioceptive, etc.) can be mounted as independent modules onto a VLA model without requiring large-scale modifications to the base architecture;
• Heterogeneous Compatibility: Each module can employ differentiated encoding strategies tailored to the data characteristics (frequency, dimensionality, temporal properties, etc.) of its respective sensory signal;
• Complementary Enhancement: Information from multiple sensory streams is effectively fused within the model, capturing complementary relationships between different signals and boosting overall perceptual capability.

The key advantage of this modular architecture lies in its scalability — when a new sensor type emerges, researchers need only design the corresponding sensory stream module and plug it into the framework, rather than retraining the entire model.

Technical Significance: A Closed-Loop Upgrade from Perception to Action

Advancing the VLA Model Paradigm

VLA models are one of the core paradigms in embodied intelligence today, unifying visual perception, language understanding, and action execution within a single end-to-end framework. However, the "V" (Vision) has long held a dominant position; while the language and action modules are powerful, their decision-making foundations remain heavily dependent on visual input. The arrival of MoSS signifies a shift in the perceptual front-end of VLA models from "monocular vision" to "multi-modal somatic sensing," giving models greater robustness when handling fine-grained manipulation and contact-rich tasks.

Practical Impact on Robot Manipulation

In real-world robotic applications, purely vision-based approaches face numerous challenges: occlusion creates visual blind spots, lighting changes affect perception accuracy, and deformations of flexible objects are difficult to model accurately from images alone. Physical sensory signals can compensate for precisely these shortcomings. By reducing the engineering complexity of multi-sensory fusion through its modular design, MoSS is poised to accelerate the deployment of tactile and other sensors in practical robotic systems.

Industry Context and Future Outlook

Embodied intelligence is currently in a period of rapid growth. Google's RT-series models, Stanford's Mobile ALOHA, and embodied intelligence platforms from numerous institutions in China are all pushing the capability boundaries of VLA models. At the same time, tactile sensor technology is maturing rapidly, with high-resolution tactile sensors such as GelSight and DIGIT providing the hardware foundation for multi-sensory fusion.

MoSS's research direction aligns closely with this broader trend. In the future, as sensor costs continue to decline and data collection efficiency improves, multi-sensory fusion is expected to move from the laboratory to large-scale deployment in real-world scenarios. It is foreseeable that the next generation of embodied agents will no longer just "see the world" but will truly "touch the world."

Notably, the modular design philosophy also facilitates cross-laboratory collaboration and open-source ecosystem development — different teams can focus on developing the sensory modules in which they specialize, ultimately achieving integration through standardized interfaces. This could accelerate technological iteration across the entire field.