Co-Evolving Policy Distillation: Cracking the Multi-Capability Fusion Challenge

Introduction: The Core Contradiction Facing Post-Training Paradigms

During the post-training phase of large language models, how to efficiently integrate the capabilities of multiple expert models into a single model has been a core challenge for both academia and industry. A recent paper published on arXiv (arXiv:2604.27083) introduces the Co-Evolving Policy Distillation (CoPD) framework, which provides a unified analysis of the two mainstream post-training paradigms — RLVR (Reinforcement Learning with Verifiable Rewards) and OPD (Online Policy Distillation). The study reveals fundamental flaws in both approaches when it comes to multi-capability fusion and proposes an innovative solution.

Key Findings: Capability Loss Mechanisms in Two Paradigms

RLVR's "Inter-Capability Divergence Cost"

RLVR has become one of the standard paradigms for LLM post-training, with its core approach being reinforcement learning training through verifiable reward signals. However, researchers found that when attempting to inject multiple expert capabilities into a single model through mixed RLVR training, an "inter-capability divergence cost" emerges between different capabilities. In simple terms, different capabilities such as mathematical reasoning, code generation, and creative writing may conflict in their optimization directions. Mixed training causes these capabilities to pull against each other, ultimately preventing the model from reaching the performance level of the corresponding expert model in any single capability.

The "Behavioral Gap" Problem in OPD Pipelines

Another common strategy is to first train expert models separately for each domain, then distill their knowledge into a unified student model through OPD. While this pipeline approach avoids the inter-capability divergence issue, it faces another bottleneck — a significant "behavioral gap" between the teacher models and the student model. Because the behavioral distributions of expert models differ too greatly from the student model, the student struggles to fully absorb all of the teachers' capabilities, significantly diminishing the distillation effectiveness.

Technical Analysis: The Innovation of Co-Evolution

The paper's key contribution lies in providing a unified theoretical framework to analyze capability loss in both paradigms and proposing the Co-Evolving Policy Distillation solution accordingly.

The core ideas of the framework can be summarized as follows:

• Unified Perspective: Incorporating RLVR and OPD into a single analytical framework to reveal the essential sources of capability loss, rather than treating them as independent technical approaches
• Co-Evolution Mechanism: Unlike the traditional serial pipeline of "train experts first, then distill," CoPD allows expert models and the unified model to evolve collaboratively during training, progressively narrowing the behavioral gap
• Dynamic Balancing Strategy: Introducing dynamic adjustment mechanisms during multi-capability optimization to mitigate divergence issues between different capabilities

From a technical standpoint, this work addresses a long-standing fundamental contradiction in LLM post-training: the balance between specialization and generalization. Single-capability expert models are easy to train but difficult to integrate, while joint training of general-purpose models faces optimization conflicts. CoPD attempts to find a viable middle path between these two extremes.

Industry Impact and Significance

This research offers important guidance for current LLM development practices:

1. Optimizing Post-Training Workflows: Many teams currently adopt "RLVR first, then distillation" or "mixed RLVR" approaches during post-training. This study clearly identifies the theoretical flaws in these methods, helping practitioners make better technical decisions
2. Reducing Training Costs: By minimizing capability loss through co-evolution, better multi-capability integration can be achieved with fewer computational resources
3. Driving Paradigm Convergence: Transforming RLVR and policy distillation from competing technical approaches into complementary solutions that can work synergistically

Outlook: The Next Step for Post-Training Technology

As the capability boundaries of large models continue to expand, multi-capability integration during the post-training phase will become increasingly important. From the GPT series to Claude, Qwen, and other leading models, maintaining top-tier performance simultaneously across mathematics, coding, multilingual support, long-context handling, and other dimensions is a common challenge faced by all model developers.

The unified analytical framework and collaborative training approach proposed by Co-Evolving Policy Distillation provides a new theoretical foundation for solving this challenge. In the future, we are likely to see more post-training approaches that deeply integrate reinforcement learning with knowledge distillation, driving large models to continuously evolve from "single-domain champions" to "all-around performers."

Notably, this research direction also forms an interesting parallel with the currently hot-debated "Model Merging" technique in the industry — both explore how to achieve multi-capability fusion at minimal cost, though they differ in their technical approaches. As theoretical research deepens, the convergence of these different directions may give rise to even more efficient post-training paradigms.