ETH Zurich Builds Neural Net That Learns Physics From Video

ETH Zurich researchers have unveiled a groundbreaking neural network architecture that can learn complex physics — including fluid dynamics, material deformation, and multi-body interactions — directly from raw video data. The system represents a significant departure from traditional physics simulation, which relies on hand-crafted equations and domain expertise, potentially accelerating scientific discovery across multiple disciplines.

Unlike previous approaches that required structured sensor data or pre-defined physical models, this new framework observes raw pixel-level video and autonomously infers the underlying physical laws governing the observed phenomena. The research positions itself at the intersection of computer vision, physics-informed machine learning, and scientific computing.

Key Takeaways

• What: A neural network that extracts physical laws from video without prior knowledge of the governing equations
• Who: Researchers at ETH Zurich, one of Europe's top technical universities
• Why it matters: Eliminates the need for hand-crafted physics models, democratizing simulation capabilities
• Technical approach: Combines differentiable rendering, graph neural networks, and latent physics representations
• Performance: Demonstrated accuracy on par with traditional numerical solvers in tested scenarios
• Applications: Engineering design, robotics, climate modeling, and materials science

How the System Learns Physics Without Equations

Traditional physics simulation requires scientists to first derive mathematical equations — often partial differential equations — that describe a system's behavior. This process can take years of theoretical work and is limited by human intuition about which variables matter.

ETH Zurich's approach flips this paradigm entirely. The neural network ingests video sequences showing physical phenomena and uses a combination of differentiable rendering and latent space dynamics to build an internal model of the physics at play. The system effectively 'watches' a ball bounce, fluid flow, or material stretch, then constructs a predictive model that can generalize to new scenarios.

The architecture employs a two-stage pipeline. First, a visual encoder extracts spatial and temporal features from video frames, converting pixel data into a structured latent representation. Second, a graph neural network (GNN) operates on this latent space to model interactions between identified physical entities — particles, rigid bodies, or mesh elements.

Graph Neural Networks Power Physical Reasoning

The choice of graph neural networks is particularly significant. GNNs naturally represent relationships between objects, making them ideal for modeling physical interactions like collisions, gravitational attraction, and elastic forces. Each node in the graph represents a physical entity, while edges encode the interactions between them.

This architecture draws on earlier work by DeepMind, whose 2020 paper on 'Learning to Simulate Complex Physics with Graph Networks' demonstrated GNNs' potential for physics simulation. However, DeepMind's approach required structured particle data as input — not raw video. ETH Zurich's contribution bridges the gap between raw visual observation and structured physical reasoning.

The system also incorporates physics-informed inductive biases, such as conservation of energy and momentum, into its loss function. These constraints ensure that learned dynamics respect fundamental physical principles, even when the network has never been explicitly told about them.

Key technical innovations include:

• A differentiable renderer that enables end-to-end training from pixel-level supervision
• Adaptive graph construction that automatically identifies interacting entities in video
• Multi-scale temporal attention for capturing both fast and slow physical processes
• A conservation-aware loss function that enforces physical plausibility
• Transfer learning capabilities that allow pre-training on simple scenarios before tackling complex ones

Benchmark Results Show Competitive Accuracy

The ETH Zurich team evaluated their system across several challenging physical scenarios, comparing performance against both traditional numerical solvers and existing machine learning baselines. The results are striking.

In fluid dynamics tests, the neural network predicted fluid behavior with less than 5% error compared to ground-truth simulations generated by established solvers like OpenFOAM. For rigid body dynamics involving multiple colliding objects, the system achieved trajectory prediction accuracy within 3% of analytical solutions.

Perhaps most impressively, the network demonstrated zero-shot generalization — the ability to predict physics in scenarios it had never seen during training. When trained on videos of 2-body collisions, it could accurately predict the outcomes of 5-body interactions. This suggests the network is truly learning underlying physical principles rather than memorizing specific scenarios.

Compared to NVIDIA's FourCastNet and other AI-driven simulation tools, the ETH Zurich system offers a unique advantage: it requires no domain-specific preprocessing. While FourCastNet needs carefully structured atmospheric data, this new approach works from video alone, making it accessible to researchers who lack extensive computational physics expertise.

Practical Applications Span Multiple Industries

The implications of learning physics directly from video extend far beyond academic curiosity. Several industries stand to benefit immediately from this technology.

Robotics represents perhaps the most direct application. Robots operating in unstructured environments need to predict how objects will behave when pushed, pulled, or dropped. Currently, this requires painstaking physics modeling for each new object and material. A video-trained physics engine could allow robots to 'learn' material properties simply by observing objects in their environment.

Manufacturing and engineering firms spend billions annually on computational fluid dynamics (CFD) and finite element analysis (FEA) simulations. These simulations require specialized software licenses costing $20,000-$100,000 per seat and expert operators. A system that learns physics from factory floor video footage could dramatically reduce both costs and time-to-insight.

Other promising applications include:

• Climate science: Learning atmospheric dynamics from satellite video to improve weather forecasting
• Medical imaging: Understanding tissue mechanics from ultrasound or MRI video sequences
• Autonomous vehicles: Predicting pedestrian and vehicle motion physics in real-time
• Game development: Automatically generating realistic physics engines from reference footage
• Materials science: Characterizing new materials by observing their deformation behavior on video

The Broader AI-for-Science Movement Gains Momentum

ETH Zurich's work fits into a rapidly growing trend of applying AI to accelerate scientific discovery. In the past 2 years alone, the field has seen transformative breakthroughs including Google DeepMind's AlphaFold for protein structure prediction, Microsoft's MatterGen for materials discovery, and Meta's ESMFold for biological sequence analysis.

What distinguishes this latest research is its generality. While AlphaFold solves one specific scientific problem brilliantly, ETH Zurich's video-based physics learning could potentially apply to any physical system that can be captured on camera. This domain-agnostic quality makes it a potential 'foundation model' for physics — analogous to how large language models serve as general-purpose text reasoning engines.

The research community has taken notice. Yann LeCun, Meta's chief AI scientist, has long advocated for AI systems that learn world models from observation, calling it a prerequisite for achieving human-level intelligence. ETH Zurich's work represents a concrete step toward this vision, demonstrating that neural networks can indeed extract structured physical knowledge from unstructured visual data.

Challenges and Limitations Remain

Despite the promising results, several significant challenges must be addressed before this technology reaches production readiness.

Computational cost remains substantial. Training the full pipeline on a single physical scenario requires approximately 200 GPU-hours on NVIDIA A100 hardware, translating to roughly $600-$800 in cloud computing costs. Scaling to more complex, real-world scenarios will demand even greater resources.

Data quality poses another challenge. The current system was primarily validated on synthetic video generated by physics engines — a somewhat circular approach. Real-world video introduces noise, occlusion, variable lighting, and camera distortion that could degrade performance. The team acknowledges that bridging the sim-to-real gap remains an active area of investigation.

There are also questions about interpretability. While the network learns accurate predictive models, the internal representations may not correspond to human-interpretable physical quantities like force, mass, or viscosity. For scientific discovery, understanding 'why' a prediction is correct matters as much as the prediction itself.

Looking Ahead: From Lab Demos to Real-World Impact

The ETH Zurich team has outlined an ambitious roadmap for future development. Near-term plans include testing the system on real-world video captured from laboratory experiments, moving beyond synthetic data. The researchers also aim to incorporate 3D reasoning from multi-view camera setups, enabling the system to learn volumetric physics rather than being limited to 2D projections.

Longer-term, the team envisions a 'universal physics engine' — a single pre-trained model capable of understanding and predicting any physical phenomenon from video observation. Such a system could fundamentally change how science is conducted, enabling researchers to rapidly hypothesize and test physical models without writing a single equation.

For developers and engineers interested in this space, the intersection of computer vision and physics simulation represents one of the most promising frontiers in applied AI. As foundation models continue to grow in capability and training data becomes more abundant, expect video-based physics learning to transition from academic novelty to industrial tool within the next 3-5 years.

The research underscores a broader truth about the current AI era: the most transformative applications may not come from making chatbots more eloquent, but from teaching machines to understand the physical world as intuitively as humans do.