Language Models Cannot Be Truly Random: 'Randomness Floor' Research Reveals Intrinsic Bias

The 'Randomness Illusion' of Language Models Is Shattered

When we ask a large language model to "just say something random," is its response truly random? A new paper numbered 2604.22771 on arXiv provides a definitive negative answer — language models are fundamentally incapable of achieving true randomness. The study, titled "The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions," is the first to systematically quantify the degree of inherent non-randomness in language models and proposes a precise measurement tool for this phenomenon.

Core Contribution: The Entropic Deviation (ED) Metric

The research team introduced a new metric called "Entropic Deviation" (ED). The metric is defined as the normalized KL divergence between a model's token distribution and a uniform distribution. In simple terms, if a model were completely random when generating the next token, every token in the vocabulary should have an equal probability of being selected; ED measures the gap between a model's actual output distribution and this "ideal random state."

KL divergence (Kullback-Leibler divergence) is a classic method in information theory for measuring the difference between two probability distributions. After normalization by the researchers, comparisons across different models and vocabulary sizes become possible. A higher ED value indicates that the model's output deviates more from a random distribution, signifying more pronounced intrinsic bias.

Experimental Design: Large-Scale Systematic Measurement

The scale of this study's experiments is impressive. The research team achieved comprehensive coverage across the following dimensions:

• Generation scale: A total of 31,200 independent generations
• Number of models: Spanning 7 different language models
• Architecture types: Covering both Transformer and State Space Model (SSM) architectures
• Prompt categories: 9 different prompt categories were designed
• Temperature parameters: 3 different temperature settings were tested
• Language coverage: 5 different natural languages were included

Particularly noteworthy is that the researchers carefully designed "semantically neutral prompts" as a key control condition. These prompts included empty strings, random character sequences, and nonsensical syllable combinations. The design intent behind these prompts was to determine whether model outputs can approach a truly random state when the input contains no meaningful semantic information.

Key Finding: The Ineliminable 'Randomness Floor'

The results revealed a profound phenomenon — even when faced with completely meaningless input, language model outputs still exhibit significant non-randomness. This non-randomness does not stem from semantic guidance in the prompts but is an inherent structural characteristic of the models themselves. The researchers aptly termed this phenomenon the "Randomness Floor" — a lower bound on randomness that models cannot break through.

This finding transcends different model architectures. Whether mainstream Transformer-based large language models or the recently emerging State Space Models, all exhibited similar patterns of intrinsic non-randomness. This suggests that the phenomenon is not a defect of any particular architecture but rather a fundamental property of the current language modeling paradigm.

While adjusting the temperature parameter can influence the "flatness" of the output distribution to some extent, models still cannot achieve a truly uniform random distribution even at higher temperature settings. This further confirms the intrinsic nature of the non-randomness — it cannot be eliminated through simple hyperparameter tuning.

Technical Analysis: Why Can't Language Models Be Random?

From a technical perspective, there are multiple reasons why language models cannot achieve true randomness:

Training data bias: Language models are trained on massive text corpora, and natural language itself is highly structured and non-uniform. Certain token combinations appear far more frequently than others, and these statistical patterns are deeply embedded in model weights.

Architectural inductive bias: Whether it is the attention mechanism in Transformers or the recurrent structures in State Space Models, specific inductive biases are introduced. These architectural designs inherently tend to produce structured rather than randomized outputs.

Properties of the softmax function: Language models typically use the softmax function to convert logits into probability distributions. Even when differences in logits are small, softmax amplifies these differences, causing the output distribution to deviate from uniformity.

Geometric structure of embedding space: Token embeddings are not uniformly distributed in high-dimensional space, and the positions of certain tokens in the embedding space make them more likely to be activated in specific contexts.

Insights from a Multilingual Perspective

The study tested across 5 languages, enabling researchers to observe whether non-randomness varies across languages. Different languages have distinct vocabulary structures, grammatical rules, and proportions in training data, all of which may influence a model's ED values for that language.

This multilingual dimension of analysis holds important reference value for understanding the fairness and consistency of language models. If models exhibit higher non-randomness in certain languages, it may indicate systematic bias arising from imbalanced training data distributions.

Research Significance and Industry Impact

The significance of this research extends beyond pure academic exploration and has practical implications for multiple application domains:

Safety and alignment: Understanding a model's intrinsic bias is crucial for AI safety research. If models still exhibit systematic tendencies when asked to generate content randomly, these tendencies may influence output quality in more subtle ways in more complex scenarios.

Random number generation: Some have attempted to use language models as pseudo-random number generators; this study clearly identifies the fundamental flaw in such approaches.

Model evaluation: The ED metric provides a new dimension for model evaluation. It can serve as a standardized tool for measuring the degree of intrinsic model bias, helping researchers make more informed decisions in model selection and optimization.

Interpretability: By quantifying the degree of model non-randomness, researchers can better understand models' internal representations and decision-making mechanisms.

Outlook: From Measurement to Governance

The discovery of the "Randomness Floor" opens up multiple directions for future research. First, how to lower this floor — that is, developing model architectures capable of more closely approximating true randomness when needed — is a topic worth exploring. Second, the ED metric itself can be further extended, for example, with variants customized for specific semantic domains or task types.

A deeper question is: do we actually need language models to possess randomness? In most application scenarios, structured and meaningful output is precisely what users expect. However, in scenarios requiring diversity, creativity, or fair sampling, understanding and managing a model's intrinsic bias becomes critically important.

This research reminds us that language models are not neutral text generators — they carry deep preferences imparted by their training data and architectural design. Recognizing this is a prerequisite for building more reliable and transparent AI systems.