Survey: How Robots Learn Manipulation Skills from Human Videos

The Urgent Need to Break the Robot Data Bottleneck

The further development of embodied intelligence and robotics faces a core bottleneck — scalable collection of robot manipulation data is extremely difficult. Unlike the natural language processing field, which can harvest massive text data from the internet, robotics data typically needs to be collected one demonstration at a time through teleoperation or kinesthetic teaching in real physical environments, making it costly and inefficient.

Recently, a survey paper published on arXiv, titled "Robot Learning from Human Videos: A Survey" (arXiv:2604.27621v1), systematically reviews a highly promising research direction: enabling robots to passively learn manipulation skills from massive human activity videos. This direction is attracting growing attention from researchers and could fundamentally transform the paradigm of robot skill acquisition.

Why Human Videos?

Hundreds of millions of human activity videos exist on the internet — from cooking tutorials to furniture assembly, from industrial operations to everyday household chores. These videos contain rich manipulation knowledge: how to grasp objects, how to use tools, the execution order of tasks, and more. If robots could effectively extract transferable skills from these videos, they would essentially have access to a nearly infinite "data goldmine."

The survey notes that the rapid emergence of this research direction is driven by two major forces:

• The abundance of human activity video data: Platforms like YouTube and TikTok generate massive amounts of video content containing manipulation behaviors daily, covering an extremely wide range of task scenarios
• Rapid advances in computer vision: Breakthroughs in large-scale visual foundation models, video understanding, 3D reconstruction, and pose estimation have provided the technical foundation for extracting robot-usable information from videos

Core Technical Challenges and Research Pathways

Transferring skills from human videos to robots is far from straightforward, with multiple levels of "gaps" that need to be bridged:

Embodiment Gap

There are significant differences between the human hand structure and robot end-effectors (such as parallel grippers and dexterous hands). How to map human hand manipulation actions into executable control commands for robots is one of the most central challenges in the field. Researchers have explored various approaches, including keypoint-based representation transfer, contact-state-based abstract representations, and using visual foundation models to extract embodiment-agnostic universal features.

Viewpoint Gap

Human-captured videos typically come from a third-person perspective, while robots usually rely on wrist-mounted cameras or fixed workstation cameras for perception. The difference in viewpoints leads to discrepancies in spatial relationship understanding. Researchers address this issue through viewpoint-invariant representation learning and multi-view data augmentation.

Action Extraction

Human videos typically contain only visual observation information and lack precise action annotations such as joint angles or end-effector poses that robots require. How to infer executable action sequences from pure visual observations is another key technical challenge. Inverse dynamics models, visuomotor policy learning, and other methods have made significant progress in this area.

Deep Integration with the Large Model Era

The survey also pays special attention to the trend of cross-pollination between this field and large-scale pretrained models. With the rapid development of vision-language models (VLMs) and video generation models, researchers have begun exploring:

• Using large language models to understand task semantics and step decomposition in videos
• Leveraging video generation models to synthesize manipulation videos from the robot's perspective as an intermediate bridge
• Employing visual foundation models to extract cross-domain universal object and scene representations

These methods are pushing the "learning from human videos" direction to new heights, moving beyond simple action imitation toward high-level task understanding and skill generalization.

Future Outlook

The survey paints an exciting picture: future robots may no longer need to be taught one task at a time through hands-on demonstration, but instead could autonomously acquire various manipulation skills by "watching" large volumes of videos, much like humans do. Once this passive learning paradigm matures, it will greatly accelerate the deployment of general-purpose robots.

However, the researchers also candidly acknowledge that the field is still in its early exploratory stage, with many open problems remaining in areas such as generalization to real-world scenarios, learning long-horizon complex tasks, and safety assurance. But as visual foundation models continue to improve and robot hardware continues to evolve, learning from human videos is poised to become the key to solving the robot data scalability challenge.

This survey provides researchers in the embodied intelligence field with a comprehensive literature map and offers important reference for industry stakeholders seeking to understand the frontier trends in robot skill learning.