Transformers Learn to Crack Pseudorandom Numbers

Transformer models can learn to predict outputs of pseudorandom number generators (PRNGs), a finding that bridges deep learning and cryptography in ways researchers are only beginning to understand. The discovery challenges long-held assumptions about the apparent randomness of algorithmic number sequences and opens new avenues for both AI research and security analysis.

This line of inquiry matters because PRNGs underpin everything from encryption protocols to Monte Carlo simulations. If a neural network can detect structure in supposedly random sequences, the implications ripple across cybersecurity, scientific computing, and our fundamental understanding of what transformers actually learn.

What Are Pseudorandom Number Generators?

Pseudorandom number generators are deterministic algorithms that produce sequences of numbers designed to appear statistically random. Unlike true random number generators, which rely on physical phenomena like atmospheric noise, PRNGs use mathematical formulas seeded with an initial value.

Popular PRNGs include:

• Linear Congruential Generators (LCGs) — among the oldest and simplest, used in legacy systems
• Mersenne Twister — the default PRNG in Python, R, and many programming languages
• Xorshift generators — lightweight alternatives popular in game development
• Linear Feedback Shift Registers (LFSRs) — widely used in hardware and stream ciphers
• Cryptographically secure PRNGs (CSPRNGs) — hardened variants used in encryption

The key property of a good PRNG is that its output should be computationally indistinguishable from true randomness — at least for any efficient algorithm. The question now is whether transformers qualify as the kind of 'efficient algorithm' that can break this indistinguishability.

How Transformers Approach the Problem

Sequence modeling is the transformer's native strength, and PRNG output is fundamentally a sequence. Researchers have explored feeding raw PRNG output tokens into transformer architectures and training them to predict subsequent values, essentially treating pseudorandom number prediction as a next-token prediction task.

The approach mirrors how large language models like GPT-4 or Claude learn to predict the next word in a sentence. Instead of natural language, the 'vocabulary' consists of numerical tokens representing PRNG outputs. The transformer's self-attention mechanism searches for long-range dependencies in the sequence — precisely the kind of hidden structure that PRNGs try to conceal.

Training typically involves generating millions of output sequences from a target PRNG, tokenizing the numerical values, and fine-tuning a transformer to minimize prediction error. The model receives no information about the PRNG's internal algorithm or state — it works purely from observed outputs.

Key Findings From Recent Experiments

Research in this area has produced several striking results. Transformers demonstrate a clear ability to learn the internal mechanics of simpler PRNGs after sufficient training.

For LCGs, transformers can often recover near-perfect prediction accuracy. This is perhaps unsurprising given that LCGs have well-known mathematical weaknesses, but the fact that a neural network discovers these weaknesses without explicit mathematical programming is notable.

With LFSRs, transformers show strong performance as well. The linear algebraic structure underlying LFSRs maps well onto the matrix operations inherent in transformer attention layers. Some experiments report that transformers effectively learn to perform the equivalent of the Berlekamp-Massey algorithm, which computes the minimal polynomial of a linear recurrence — a remarkable emergent capability.

Mersenne Twister presents a more complex challenge due to its 624-dimensional state space. However, researchers have shown that with enough observed outputs (typically 624 or more consecutive values), transformers can begin to model its behavior, mirroring known mathematical attacks that exploit the generator's linear recurrence over GF(2).

Why This Matters for Cryptography

Cryptographic security relies on the assumption that no efficient algorithm can distinguish PRNG output from true randomness. If transformers can learn to break this assumption for certain generators, it has direct security implications.

The good news for cryptographers is that CSPRNGs like ChaCha20 or AES-CTR remain far beyond what current transformers can crack. These generators are designed to resist even adversaries with enormous computational budgets, and nothing in current research suggests transformers pose a threat to them.

However, the research highlights risks in 3 key areas:

1. Legacy systems still using weak PRNGs like LCGs could be more vulnerable than previously assumed, especially if attackers deploy AI-based analysis
2. Custom or proprietary PRNGs used in gaming, lottery systems, or IoT devices may have exploitable patterns that transformers can detect
3. Side-channel analysis could benefit from transformer-based approaches that identify subtle statistical biases in supposedly secure implementations

What Transformers Reveal About Their Own Learning

Perhaps the most fascinating aspect of this research is what it reveals about transformer internals. When a transformer learns to predict PRNG outputs, it must develop internal representations of the generator's mathematical structure.

Mechanistic interpretability researchers have begun probing these trained models to understand what algorithms the attention heads implement. Early findings suggest that transformers develop specialized circuits — some heads perform modular arithmetic, others track state bits, and still others implement the equivalent of polynomial division.

This connects to broader work on grokking, the phenomenon where neural networks suddenly transition from memorization to generalization on algorithmic tasks. PRNG prediction serves as an ideal testbed because the underlying algorithm is known, making it possible to verify whether the transformer has truly learned the generating process or merely memorized patterns.

Limitations and Open Questions

Scalability remains the primary limitation. As PRNG state spaces grow and algorithms become more complex, the training data and model capacity required increase dramatically. Current experiments work well on generators with state spaces up to a few hundred bits, but scaling to thousands of bits — typical for cryptographic applications — is not yet feasible.

Several open questions persist:

• Can transformers learn to break nonlinear PRNGs that resist traditional algebraic attacks?
• What is the minimum number of observed outputs needed for a transformer to learn a given generator?
• Do larger models like those with billions of parameters offer qualitative advantages, or do diminishing returns set in quickly?
• Could hybrid approaches combining transformers with symbolic reasoning crack generators that neither method solves alone?

Looking Ahead: Implications for AI and Security

Future research is likely to move in 2 directions simultaneously. On the AI side, PRNG learning serves as a controlled laboratory for understanding how transformers develop mathematical reasoning — a question central to building more capable AI systems.

On the security side, this work motivates renewed scrutiny of PRNG implementations across industries. Organizations relying on non-cryptographic PRNGs for any security-sensitive application should treat this research as an early warning.

The intersection of deep learning and cryptanalysis is still in its infancy, but the trajectory is clear. As transformer architectures grow more powerful and training techniques more sophisticated, their ability to find structure in seemingly random data will only improve. For now, the strongest PRNGs remain safe — but the margin of safety deserves constant re-evaluation in an era where AI capabilities advance quarterly rather than yearly.