The 'Spectral Lifecycle' of Transformer Training Systematically Revealed for the First Time

Introduction: Peering Into the 'Black Box' of Transformer Training

The training process of large language models has long been regarded as an opaque black box. Although we can observe the decline of loss curves and the improvement of final performance, there has been a persistent lack of systematic research into what kind of structural evolution the internal weight matrices actually undergo. A recent paper published on arXiv (arXiv:2604.22778v1) has, for the first time, conducted a comprehensive longitudinal tracking of the singular value spectra of weight matrices during Transformer pre-training, revealing three previously unknown training dynamics phenomena and opening a new window into understanding the intrinsic learning mechanisms of large models.

Core Findings: Three Major Spectral Evolution Phenomena

Phenomenon 1: Transient Compression Waves

The research team performed full SVD (Singular Value Decomposition) on every weight matrix at intervals of every 25 steps across three model scales ranging from 30M to 285M parameters. They discovered a remarkably striking phenomenon — 'Transient Compression Waves.'

Specifically, the stable rank compression of weight matrices does not occur simultaneously across all layers. Instead, it propagates in the form of a 'traveling wave' from earlier layers to later layers. This process creates a notable gradient effect: the degree of compression peaks in the early stages of training and then gradually diffuses toward deeper layers of the network. This means that during the Transformer's learning process, low-rank compression of information has a clear directionality and temporal ordering, like a wave advancing layer by layer from the input end to the output end.

This finding overturns the previously implicit assumption that all layers 'learn synchronously,' demonstrating that layers at different depths play distinctly different roles at different stages of training.

Phenomenon 2: Persistent Spectral Gradients

The second key finding is 'Persistent Spectral Gradients.' Unlike transient compression waves, spectral gradients are a structural feature that persists throughout the training process. The research shows that from shallow to deep layers, the spectral distribution of weight matrices exhibits systematic differences, forming a sustained gradient pattern.

The existence of this spectral gradient hints at an important architectural property: there are fundamental differences in the functional specialization of different layers in a Transformer. Shallow layers may tend to preserve the diversity of input information, while deeper layers tend to compress information into lower-dimensional subspaces. This division of labor is not predetermined but emerges naturally during the training process.

Phenomenon 3: Q/K-V Asymmetry

The third finding is perhaps the most practically significant — a notable spectral asymmetry exists between the Query/Key matrices and the Value matrices in the attention mechanism (Q/K-V Asymmetry).

In the standard Transformer architecture, the Q, K, and V projection matrices are typically treated as symmetric components, often sharing the same configuration for initialization and optimization strategies. However, SVD analysis clearly reveals that Q/K matrices and V matrices follow distinctly different spectral evolution paths during training. Q and K matrices tend to develop more concentrated spectral structures, while V matrices maintain a relatively dispersed singular value distribution.

This asymmetry makes intuitive sense from an information-theoretic perspective: Q/K are responsible for computing attention weights, which is essentially a process of 'matching' and 'selection,' well-suited to capturing key patterns with low-rank structures. The V matrix, on the other hand, is responsible for carrying and transmitting actual semantic information, requiring richer representational dimensions.

Technical Analysis: A Methodological Breakthrough

Unprecedented Observation Granularity

The methodological breakthrough of this study lies in its observation granularity. Performing full SVD on all weight matrices at 25-step intervals means the researchers obtained an extremely fine-grained 'training electrocardiogram.' Most prior related work focused only on weight matrix characteristics after training concluded, or analyzed at only a few checkpoints. This dense sampling frequency allowed transient phenomena like the 'traveling wave' to be captured for the first time.

Cross-Scale Consistency

The observation of consistent phenomena across three parameter scales — 30M, approximately 100M, and 285M — strengthens the generalizability of the findings. Although the current experimental scales have not yet covered ultra-large models with billions or even hundreds of billions of parameters, the consistency across three orders of magnitude provides a reasonable basis for expecting broader applicability.

Complementing and Challenging Existing Theories

These findings form interesting parallels with recent theories on 'Neural Collapse,' 'Low-Rank Adaptation' (the theoretical basis of LoRA), and 'progressive feature learning' in neural network training. The existence of transient compression waves provides new theoretical support for parameter-efficient fine-tuning methods like LoRA — if weight matrices inherently tend toward low-rank structures during training, then the rationale for fine-tuning within low-rank subspaces is further validated.

At the same time, the discovery of Q/K-V asymmetry calls into question current 'one-size-fits-all' initialization and regularization strategies. If different types of weight matrices follow different dynamical laws during training, then designing differentiated training strategies tailored to each type could potentially yield performance improvements.

Practical Implications and Future Outlook

Training Strategy Optimization

The discovery of transient compression waves provides new design principles for learning rate scheduling and layer-wise adaptive optimization. For example, learning rates for different layers could be dynamically adjusted according to the propagation stage of the compression wave — allowing more learning headroom when the wave arrives and appropriately reducing it after compression is complete.

Model Compression and Pruning

Persistent spectral gradients point the way for structured pruning and model compression. Stronger low-rank characteristics in deeper layers indicate greater compression potential in those layers, while shallower layers may need to retain more parameters to maintain information diversity.

Architecture Design

Q/K-V asymmetry may inspire new attention mechanism designs. Future architectures might allocate different dimensions to Q/K and V or adopt different parameterization strategies to better match their respective functional requirements.

Limitations and Future Directions

The primary limitation of the current study is that the model scale caps at 285M parameters, which is still a considerable gap from today's mainstream models with billions to hundreds of billions of parameters. Additionally, the experiments only covered the standard Transformer architecture, and applicability to variants such as MoE (Mixture of Experts) and state space models remains to be verified.

Future research directions may include: extending spectral tracking to larger-scale models, exploring the relationship between compression waves and emergent capabilities, and designing adaptive training algorithms based on spectral dynamics. This work represents a solid step toward 'opening the training black box,' and its methodology and findings will provide important baseline references for subsequent research.

Conclusion

The value of this paper lies not only in its specific findings but also in pioneering a paradigm for systematically studying training dynamics. Just as astronomers understand the lifecycle of stars through spectral analysis, researchers can now understand the Transformer's learning process through the 'spectral lifecycle' of weight matrices. This may mark yet another important milestone in the evolution of large model research from 'alchemy' to 'precision science.'