Heterogeneous Grouped Mixture-of-Experts Architecture: A New MoE Paradigm Breaking the Uniform Expert Bottleneck

Traditional MoE Faces a One-Size-Fits-All Dilemma

Mixture-of-Experts (MoE) has become a key architecture for efficiently scaling large language model (LLM) performance in industrial applications. From Google's Switch Transformer to Mistral's Mixtral series, MoE has been widely adopted across the industry, dramatically boosting model capacity through sparse activation mechanisms without proportionally increasing computational costs.

However, standard MoE architectures suffer from a fundamental limitation — all expert networks are forced to be designed at the same size. This "uniform expert" paradigm introduces significant rigidity: regardless of how the semantic complexity of input tokens varies, the computational resources consumed by each routing activation remain fixed. Simple grammatical connectives and specialized terminology requiring deep reasoning are allocated exactly the same computational budget — clearly not an optimal resource allocation strategy.

Core Innovation: Heterogeneous Grouped Mixture-of-Experts Architecture

A recent paper published on arXiv (arXiv:2604.23108v1) introduces a new architecture called Mixture of Heterogeneous Grouped Experts, designed to fundamentally resolve this contradiction.

The core ideas of this approach can be summarized as follows:

1. Breaking the Uniformity Constraint on Expert Size

Unlike traditional MoE where all experts share the same hidden layer dimensions and parameter counts, the new architecture allows expert networks of different sizes to coexist within the same MoE layer. Large experts possess stronger expressive capacity and are suited for processing high-complexity tokens, while small experts handle simple tokens quickly with lower computational overhead.

2. Introducing a Grouping Mechanism to Optimize System Efficiency

Previous researchers have attempted heterogeneous expert approaches, but often faced severe system-level challenges — experts of different sizes are difficult to parallelize efficiently on GPUs, causing theoretical efficiency gains to be offset by practical engineering overhead. This paper employs a "Grouped" strategy, organizing experts of the same size into groups and enabling efficient batched parallel computation within each group. This significantly alleviates engineering deployment challenges while preserving heterogeneous flexibility.

3. Complexity-Aware Dynamic Routing

The routing mechanism no longer merely selects "which experts" but also implicitly decides "how much computation to allocate." When the router assigns a token to a large expert group, it essentially invests more computational resources; the reverse achieves computational savings. This mechanism enables the model to adaptively adjust computational allocation based on input content during inference.

Technical Significance and In-Depth Analysis

From "Computational Fairness" to "Computation on Demand"

Traditional MoE can be viewed as a "computational fairness" model — each token receives the same expert capacity. The heterogeneous grouped expert architecture drives a shift toward a "computation on demand" paradigm. This concept aligns closely with the increasingly prominent trend of Adaptive Computation, including directions such as Early Exit mechanisms and dynamic depth networks.

A Key Breakthrough in Engineering Feasibility

Notably, the "grouping" design in this paper is not merely theoretical packaging but a precise response to real deployment pain points. In distributed training and inference scenarios, irregular tensor shapes severely undermine hardware utilization. By grouping homogeneous experts together, computation within each group can be efficiently completed using regular matrix operations — a critical factor for practical deployment on modern GPU and TPU clusters.

Comparison with Current Mainstream Architectures

Compared to the Fine-grained Experts strategy adopted by DeepSeek-V3 — which improves flexibility by increasing the number of experts while reducing individual expert size — heterogeneous grouped experts offer optimization along an orthogonal dimension. The former innovates on "quantity," while the latter seeks breakthroughs in "size diversity." The two approaches may even be combined in future applications.

Future Outlook

The Mixture of Heterogeneous Grouped Experts architecture opens a new door for MoE evolution. As large language models continue advancing toward greater scale and higher efficiency, "teaching models how to allocate their own computational resources" is becoming one of the core themes in architecture design.

If this method proves effective in larger-scale experiments, there is good reason to expect its integration into the architecture designs of next-generation open-source or commercial large models. Meanwhile, this approach may also extend to multimodal models — tokens from different modalities naturally carry different processing complexities, and heterogeneous expert architectures could offer unique advantages in cross-modal fusion.

The evolution of MoE architectures is far from over, and the combination of "heterogeneity" and "grouping" may well be a crucial direction for the next step in that evolution.