LLMs Fail the Dice Roll: Large Language Models Struggle to Generate Statistically Random Numbers

Introduction: When AI Is Asked to Roll the Dice

We are accustomed to marveling at the astonishing capabilities of large language models (LLMs) in writing, coding, and reasoning. But what happens when they are given a seemingly simple task — generating random numbers from a probability distribution? A recent paper published on arXiv (arXiv:2601.05414v3) delivers a surprising answer: LLMs perform remarkably poorly when it comes to rolling the dice.

As LLMs evolve from mere chatbot tools into core components of stochastic pipelines and general-purpose intelligent systems, faithfully sampling from specified probability distributions is no longer a matter of theoretical curiosity — it is a critical functional requirement. The findings of this study could have far-reaching implications for the reliability and safety of AI systems.

Core Findings: 11 Frontier Models Collectively Fail

Conducted by a team of researchers, this study represents the first large-scale, statistically powered systematic audit of native probabilistic sampling capabilities in frontier LLMs. The research team benchmarked 11 mainstream large language models — including the GPT series, Claude series, and Llama series — across 15 different statistical distribution scenarios.

The results show that when asked to generate random numbers from common statistical distributions such as uniform, normal, and Poisson distributions, these models universally exhibited significant systematic biases. Specifically, the number sequences generated by the models almost invariably failed randomness tests under statistical examination, displaying clear pattern-driven tendencies.

For example, in the most basic uniform distribution sampling task — equivalent to simulating a fair die — LLMs tended to favor certain specific numbers while avoiding others. This preference was not random fluctuation but a highly reproducible systematic bias. In more complex continuous distribution sampling, model performance was even more unsatisfactory, with generated samples showing significant deviations from target distributions in shape, scale, and location parameters.

Deep Analysis: Why LLMs Struggle with Randomness

The reasons behind this phenomenon merit in-depth exploration. A large language model is fundamentally a conditional probability predictor — it predicts the next most likely token based on context. This architecture inherently tends toward generating outputs that "look reasonable" rather than truly random outputs.

Training data bias is the first key factor. The way humans use numbers in everyday text is itself far from uniformly distributed. For instance, the number 7 is frequently chosen as a "random number" in human psychology experiments, and this preference has likely been encoded into model parameters.

Limitations of the autoregressive generation mechanism represent a second deep-seated cause. The way LLMs generate numbers token by token means that the choice of a preceding number influences the generation of subsequent numbers. This makes it extremely difficult for models to produce truly independent and identically distributed random samples, with sequences often containing implicit autocorrelation structures.

The gap between semantic understanding of "randomness" and its mathematical implementation should not be overlooked either. A model may "understand" the conceptual definition of a uniform distribution, but translating that understanding into actual sampling behavior that meets mathematical requirements is an entirely different matter. This exposes a fundamental gap in current LLMs between knowledge and execution.

Real-World Impact: Far More Than an Academic Issue

The significance of this research extends well beyond academic discussion. In the current AI application ecosystem, LLMs are increasingly being embedded in complex systems that require random sampling capabilities:

• Monte Carlo simulations: Widely used in financial risk assessment and scientific computing, these methods depend on high-quality random number generation
• AI agent decision-making: When AI agents need to make exploratory decisions in uncertain environments, the quality of random sampling directly affects decision outcomes
• Synthetic data generation: When LLMs are used to generate training data conforming to specific distributions, biases propagate to downstream models
• Probabilistic programming and Bayesian inference: Emerging paradigms that use LLMs as inference engines impose strict requirements on sampling accuracy

If developers take LLM random number generation capabilities for granted without understanding these limitations, they risk introducing subtle yet far-reaching biases into their systems.

Outlook: Possible Paths to Bridging the Randomness Gap

The researchers note that addressing this problem will likely require a multi-pronged strategy. One straightforward approach is to integrate verified external random number generators into LLM systems, decoupling sampling tasks from the language model and delegating them to specialized tools. This "tool-calling" approach aligns closely with the current development trajectory of AI agents.

From a model perspective, future research could explore enhancing models' faithful sampling capabilities for probability distributions through specialized fine-tuning or reinforcement learning. Other researchers have proposed introducing calibration mechanisms during inference to post-process the numerical sequences output by models, bringing them closer to target distributions.

This study reminds us that on the road to AGI, some seemingly basic mathematical capabilities may be precisely the most easily overlooked weaknesses of large language models. As we grant AI ever-greater autonomy, understanding and honestly confronting these limitations is far more important than blindly trusting in model omnipotence. As the paper's title implies: even the most powerful AI is far from a competent player when it comes to rolling the dice.