Where Should LoRA Go? New Research Reveals Optimal Fine-Tuning Strategies for Hybrid Models

The Hybrid Architecture Era Brings New Challenges for LoRA Fine-Tuning

With the rapid rise of hybrid language models, architectures that interleave attention mechanisms with recurrent components (such as Mamba SSM and GatedDeltaNet) are fiercely competing with pure Transformer models. However, a widely overlooked question has surfaced: when we perform parameter-efficient fine-tuning on these hybrid models, where exactly should LoRA adapters be placed?

Recently, a new paper published on arXiv, "Where Should LoRA Go? Component-Type Placement in Hybrid Language Models," conducted a systematic study of this question, providing important guidance for efficient fine-tuning of hybrid architectures.

Core Finding: Uniform Placement Is Not Optimal

Current mainstream LoRA fine-tuning practices typically apply adapters uniformly across all layers of a model, without distinguishing the functional roles of different component types. However, in hybrid architectures, attention layers and recurrent layers handle fundamentally different computational tasks — attention layers excel at global information interaction, while recurrent components (such as SSMs) focus on efficient sequence modeling.

The research team selected two representative hybrid architectures for experimentation:

• Qwen3.5-0.8B: Employs a sequential hybrid design, arranging GatedDeltaNet and softmax attention mechanisms in sequence
• Falcon-H1-0.5B: Employs a parallel hybrid design, combining Mamba-2 SSM with attention mechanisms in parallel

These two architectures represent the two dominant paradigms in current hybrid model design — sequential interleaving and parallel fusion — lending strong generalizability to the research conclusions.

Technical Analysis: Component Type Determines Fine-Tuning Effectiveness

The paper's core methodology is "Component-Type LoRA Placement." Rather than simply adding LoRA to all layers, the researchers systematically explored the following strategies:

1. LoRA on attention layers only: Concentrating the entire fine-tuning parameter budget on attention components
2. LoRA on recurrent layers only: Applying adapters exclusively to recurrent modules such as SSM or GatedDeltaNet
3. Uniform placement: The traditional approach, without distinguishing component types
4. Proportionally differentiated placement: Allocating different LoRA ranks based on component characteristics

The key significance of this research lies in revealing a fact long overlooked by the community: different component types in hybrid models respond to fine-tuning signals in markedly different ways. Under the same parameter budget constraints, targeted allocation of LoRA resources can yield significant performance improvements.

Implications for Industry Practice

This research has multiple implications for the current large model fine-tuning ecosystem:

For model developers, when adapting next-generation hybrid architecture models like Qwen3.5 and Falcon-H1 to downstream tasks, they should no longer rely on the "one-size-fits-all" LoRA strategy from the pure Transformer era. Instead, they need to develop differentiated fine-tuning plans based on specific architectural characteristics.

For framework developers, mainstream fine-tuning frameworks such as PEFT and LLaMA-Factory need to add support for component-level LoRA configuration in hybrid architectures, enabling users to flexibly specify adapter parameters for different component types.

For the research community, this work opens a new research direction — as more hybrid architectures such as Jamba, Zamba, and Griffin emerge, parameter-efficient fine-tuning strategies for heterogeneous components will become a critical topic.

Future Outlook

Hybrid architectures are becoming a major development direction for large language models, with their advantages in long-sequence processing efficiency and inference speed becoming increasingly prominent. However, fine-tuning methodologies for these new architectures remain in early exploratory stages.

Although this research currently focuses on small-scale models (0.5B–0.8B parameters), its methodological framework is fully applicable to larger-scale hybrid models. As the Qwen, Falcon, and other model families continue to release larger hybrid architecture versions, the value of component-level LoRA placement strategies will be further amplified.

It is foreseeable that future parameter-efficient fine-tuning tools will evolve from being "architecture-agnostic" to "architecture-aware," automatically identifying different component types within a model and intelligently allocating fine-tuning resources. This may well become one of the core features of next-generation PEFT technology.