Fréchet Mean Redefines the Geometry of Model Merging

The Core Dilemma of Model Merging: Symmetry Has Been Overlooked

In the era of large models, how to efficiently merge multiple independently trained models into a single unified model has become a persistent hot topic in the research community. Model merging techniques aim to fuse the capabilities of multiple models into a single model without additional training — a goal of significant importance for reducing computational costs and consolidating multi-task capabilities.

However, the most naive parameter-space averaging approach suffers from a fundamental flaw: it completely ignores the inherent symmetries in neural network architectures. For example, the ordering of neurons within a layer can be arbitrarily permuted without affecting the network's function, meaning that the parameter space contains a vast number of solutions that are "functionally equivalent" yet "numerically different." Under such symmetries, simple parameter averaging can produce merged results with severely degraded performance.

A recent paper published on arXiv (arXiv:2604.27155) provides an in-depth analysis of this problem and presents a refreshing perspective: not only must the geometric structure underlying the merging process be symmetry-invariant, but the averaging operation itself must also satisfy symmetry invariance.

Revisiting Model Merging from a Geometric Perspective

Limitations of Traditional Methods

Most mainstream model merging methods today are based on linear interpolation or weighted averaging in parameter space. For instance, given two models fine-tuned on different tasks, the most straightforward approach is to compute an element-wise average of their parameters. Such methods are simple and efficient, but they implicitly assume that Euclidean distance in parameter space reasonably reflects the "functional distance" between models.

The reality is far from this assumption. Due to the extensive architectural symmetries in neural networks — including neuron permutation symmetry, scale symmetry, and others — Euclidean geometry in parameter space cannot accurately characterize the true relationships between models. Two functionally identical models may be far apart in parameter space, while two models with similar parameters may exhibit vastly different functional behaviors.

In recent years, researchers have recognized the need to perform model merging under "symmetry-aware" geometric structures. For example, methods that first align neuron permutations before averaging (such as Git Re-Basin and related work) have achieved significant progress. However, these methods often focus only on correcting the geometric structure while overlooking a deeper mathematical issue.

Key Insight: The Averaging Operation Itself Must Also Be Symmetry-Invariant

The core contribution of this paper lies in revealing a previously overlooked point: even when working in a symmetry-invariant geometric space, the merging result can still be problematic if the averaging operation itself does not satisfy symmetry invariance.

In other words, symmetry-aware model merging must simultaneously satisfy two conditions:

1. Symmetry invariance of the geometric structure: The metric used to measure distances between models must respect architectural symmetries;
2. Symmetry invariance of the averaging operation: The aggregation operation used to compute the "center point" must remain consistent under symmetry transformations.

This insight may seem simple, but its mathematical implications are profound, directly leading to the core solution proposed in the paper.

Fréchet Mean: A Universal Mathematical Framework for Model Merging

What Is the Fréchet Mean

The Fréchet mean is a natural generalization of the classical arithmetic mean to general metric spaces. For a set of data points, the Fréchet mean is defined as the point that minimizes the sum of squared distances to all data points. In Euclidean space, the Fréchet mean reduces to the ordinary arithmetic mean; but in more general geometric structures (such as Riemannian manifolds, quotient spaces, etc.), the Fréchet mean naturally respects the geometric properties of the underlying space.

The paper proposes that formalizing the model merging problem as a Fréchet mean computation can elegantly address both conditions simultaneously. Specifically:

• First, define an appropriate metric on the quotient space that accounts for architectural symmetries;
• Then, compute the Fréchet mean of the models to be merged in this metric space as the merging result.

Since the definition of the Fréchet mean inherently depends on the metric structure, when the metric itself satisfies symmetry invariance, the computed mean point automatically satisfies symmetry invariance as well. This dual consistency of "geometry + averaging" is precisely what previous methods have been missing.

Theoretical Unification

Another important advantage of this framework is its theoretical unifying power. The paper demonstrates that many existing model merging methods can be viewed as special cases of the Fréchet mean framework:

• Ordinary parameter averaging corresponds to the Fréchet mean in Euclidean space;
• Alignment-based merging methods (such as averaging after permutation search) can be understood as approximate Fréchet means in specific quotient spaces;
• More complex merging strategies can also be incorporated into the unified framework by choosing different metric structures.

This unified perspective not only helps in understanding the intrinsic connections among existing methods but also provides systematic guiding principles for designing new merging algorithms.

Technical Significance and Potential Impact

Advancing Model Merging Research

Model merging technology plays an increasingly important role in the large model ecosystem. As the open-source community sees a surge of model variants fine-tuned for different tasks, efficiently fusing the capabilities of these models has become a practical necessity. The Fréchet mean framework provides a more solid theoretical foundation for this challenge and is poised to drive progress in the following directions:

• More robust merging algorithms: Grounded in rigorous geometric theory, new algorithms may exhibit better stability when dealing with large-scale models;
• Automated merging pipelines: A unified mathematical framework facilitates the development of tools that automatically select optimal merging strategies;
• Theoretical guarantees on merging quality: The mathematical properties of the Fréchet mean may provide provable lower bounds on the quality of merged results.

Connections to Current Hot Topics

Notably, model merging is closely related to several trending directions in the large language model space. In MoE (Mixture of Experts) architectures, the fusion of different expert modules can be viewed as a form of generalized model merging; in federated learning, aggregating model updates from different clients is essentially a symmetry-aware averaging problem. The generality of the Fréchet mean framework gives it potential application value across all these scenarios.

Furthermore, as model merging becomes widely adopted in the open-source community (through tools such as "mergekit" on Hugging Face), this theoretical work is expected to offer practitioners deeper understanding and better default strategy choices.

Outlook: From Theoretical Elegance to Practical Implementation

Despite the theoretical appeal of the Fréchet mean framework, its practical application still faces several challenges. First, computing the exact Fréchet mean in high-dimensional parameter spaces may encounter computational complexity bottlenecks, especially when the metric structure of the quotient space is complex. Second, how to select the most appropriate metric structure for a specific model architecture still requires further experimental validation and theoretical analysis.

Nevertheless, the significance of this work lies more in providing a unified and rigorous thinking framework for the field of model merging. Just as Riemannian geometry provided the mathematical language for general relativity, the Fréchet mean framework has the potential to become the "common language" for understanding and designing model merging algorithms.

In an era of continuous large model iteration and a thriving open-source ecosystem, the theoretical deepening of model merging technology will undoubtedly have a profound impact on the entire AI field. The contribution of this paper represents a solid step forward on that path.