Sigmoid Attention Mechanism Revolutionizes Single-Cell Foundation Model Training

Traditional Attention Mechanisms Hit a Bottleneck in Biological Foundation Models

In the large language model domain, Softmax attention has long been the de facto standard for Transformer architectures. However, when researchers attempted to transfer this paradigm to biology — particularly for building foundation models in single-cell genomics — things didn't go as smoothly as expected. Training instability and suboptimal representation learning have been persistent challenges for the biological AI research community.

A recent paper published on arXiv (arXiv:2604.27124v1) proposes a seemingly simple yet remarkably effective solution: directly replacing Softmax attention with Sigmoid attention to achieve better representation quality and faster training speeds in single-cell foundation models.

Core Findings: Improvements in Both Representation Quality and Training Efficiency

The study's core contributions can be summarized in two key areas:

Superior Learned Representations

The research team conducted systematic evaluations across six diverse single-cell datasets, yielding impressive results. Models using Sigmoid attention achieved approximately a 25% improvement in cell type separation metrics, while also delivering better performance on cell type cohesion metrics. Additionally, the models exhibited lower validation loss, indicating that Sigmoid attention helps capture biological signals in single-cell data more accurately.

This means that foundation models trained with Sigmoid attention can more clearly distinguish between different cell types while clustering cells of the same type more tightly in the representation space — providing direct value for downstream tasks such as cell annotation and trajectory analysis.

Faster Training Speed

Beyond representation quality gains, Sigmoid attention also delivered up to a 10% training speedup. For foundation models that routinely process millions or even hundreds of millions of single-cell data points, a 10% speed improvement translates to significant computational resource savings and shorter development cycles.

Technical Deep Dive: Why Is Sigmoid Better Suited for Biological Data?

The core characteristic of Softmax attention lies in its "competitive normalization" — all attention weights must sum to 1, meaning that attending to one position necessarily comes at the cost of reduced attention to others. Sigmoid attention, by contrast, allows each position to independently receive an attention weight between 0 and 1, without participating in global competition.

In single-cell genomic data, a cell may simultaneously express multiple functionally related yet independent gene modules. Sigmoid's "non-competitive" nature enables the model to attend to multiple independent biological signals simultaneously, without being forced to make trade-offs due to normalization constraints. This property aligns more naturally with the inherent structure of biological data.

Notably, Sigmoid attention is not an entirely new concept. Previous studies have explored its application in natural language processing and computer vision. However, this research is the first to systematically validate its superiority in single-cell foundation models, demonstrating that it can serve as a "plug-and-play" replacement for Softmax without requiring major architectural modifications.

Industry Impact and Future Outlook

Single-cell foundation models represent one of the most active research directions in biological AI today. From Genentech's scGPT to related projects by the Chan Zuckerberg Initiative, the industry is investing substantial resources in building general-purpose models capable of understanding biology at the cellular level. The Sigmoid attention replacement strategy proposed in this study offers the field a low-cost, high-reward optimization pathway.

From a broader perspective, this work also provides insights into "domain adaptation" for AI foundation models: the optimal architectural choices in general NLP may not be the best solutions for vertical domains. Revisiting the design of fundamental components based on domain-specific data characteristics can yield unexpected gains.

Looking ahead, the research team may further explore how Sigmoid attention scales on larger single-cell datasets, as well as its applicability in multi-omics foundation models (such as spatial transcriptomics and epigenomics). This seemingly "minor" architectural modification could lay an important foundation for the next generation of biological AI foundation models.