PivotMerge: A New Paradigm for Model Merging in Heterogeneous Multimodal Pretraining

A New Solution for Multimodal Large Model Fusion

The capability improvement of Multimodal Large Language Models (MLLMs) heavily depends on pretraining with diverse data sources, yet different datasets often endow models with distinct cross-modal alignment abilities. How to efficiently integrate multiple expert models, each with its own strengths, into a unified model has long been a core concern in the research community. Recently, a paper published on arXiv proposed a novel method called "PivotMerge," which aims to tackle the challenge of merging heterogeneous multimodal models starting from the pretraining stage.

The Core Problem: A Gap in Pretraining-Stage Model Merging

Model merging has been proven to be a low-cost, high-efficiency approach for integrating model capabilities. Its core idea is to combine multiple "expert models" trained on different tasks or data into a single unified model that possesses the advantages of all, without retraining from scratch.

However, virtually all existing model merging research focuses on "post-finetuning" scenarios — merging parameters after supervised fine-tuning (SFT) or instruction tuning. Exploration of model merging at the pretraining stage remains nearly nonexistent. This gap presents critical challenges:

• Heterogeneity problem: Models trained on different pretraining data sources (such as image-text pairs, video-text, document OCR data, etc.) exhibit vastly different distributions in parameter space. Simple parameter averaging or linear interpolation often leads to severe performance degradation.
• Alignment conflicts: The cross-modal alignment patterns learned by different models may contradict each other, and direct merging can destroy existing alignment quality.
• Scale bottleneck: The computational cost of joint retraining is prohibitively high, creating an urgent need for more economical alternatives.

Technical Approach: The Core Ideas Behind PivotMerge

The key innovation of PivotMerge lies in introducing a "post-alignment" mechanism to bridge the parameter space gap between heterogeneous pretrained models. Specifically, the method's technical pathway can be summarized in the following key steps:

First, constructing an anchor space. PivotMerge selects a base model as the "pivot" and uses its parameter space as a unified frame of reference. The parameters of other expert models are aligned to this common space, thereby eliminating the distributional shift caused by heterogeneous pretraining.

Second, post-alignment transformation. For each expert model to be merged, PivotMerge maps its parameters to the pivot model's parameter space through a lightweight alignment transformation. This process does not require re-executing full pretraining, keeping computational overhead manageable.

Third, intelligent parameter fusion. Within the unified parameter space, PivotMerge employs targeted merging strategies that fully preserve the complementary advantages of each expert model while suppressing potential capability conflicts.

The elegance of this approach lies in transforming the seemingly irreconcilable problem of "heterogeneous model merging" into a tractable two-stage problem of "align first, then merge," significantly reducing both technical difficulty and computational cost.

Significance Analysis: Why Pretraining-Stage Merging Matters

From the perspective of technological development, PivotMerge's value is reflected on multiple levels:

Reducing training costs. Current mainstream MLLM pretraining routinely consumes millions of GPU hours. If independently pretrained models from different data sources can be directly merged rather than jointly trained, it would yield orders-of-magnitude savings in computational resources.

Fostering a collaborative model ecosystem. Different teams can each focus on pretraining for specific modalities or data domains, ultimately integrating capabilities through PivotMerge-like methods to form a new collaborative paradigm of "distributed pretraining, centralized merging."

Expanding the theoretical boundaries of model merging. Previously, the theoretical foundations of model merging were primarily built on locality assumptions in the parameter space of fine-tuned models. PivotMerge advances research into the pretraining stage, where parameter space differences are much larger, providing a new experimental foundation for deepening model merging theory.

Industry Context and Related Progress

Model merging technology has developed rapidly in recent years. From early simple parameter averaging to the successive introduction of methods such as Task Arithmetic, TIES-Merging, and DARE, the precision and applicability of model merging have continuously expanded. Meanwhile, in the multimodal domain, the rapid iteration of representative models such as LLaVA, Qwen-VL, and InternVL has also provided rich application scenarios for model merging technology.

Notably, previous researchers have attempted model merging in multimodal scenarios, but most efforts were limited to homogeneous models sharing the same pretrained base. PivotMerge explicitly targets "heterogeneous" scenarios, taking an important step forward in problem definition.

Future Outlook

PivotMerge offers a highly promising new path for the efficient construction of multimodal large models. Looking ahead, the following directions deserve continued attention:

• Extending to more modalities: Current work primarily revolves around vision-language modalities, with potential future expansion to audio, 3D, sensor data, and other broader modality combinations.
• Integration with continual learning: Model merging is naturally suited for incremental capability expansion, and combining it with continual learning paradigms could give rise to more flexible model evolution strategies.
• Theoretical guarantees for merging quality: How to theoretically characterize the performance bounds of merged models remains an open and important fundamental question.

As multimodal AI systems evolve toward larger scales and more modalities, efficient model integration technologies like PivotMerge are poised to become critical infrastructure in the next-generation MLLM construction pipeline.