DeepSeek V4's Biggest Regret: Where Is Engram?

DeepSeek V4 Launches Without Its Most Anticipated Feature

DeepSeek V4 arrived with a packed technical report — featuring mHC, CSA, HCA, Muon optimization, and FP4 quantization — yet the one innovation the AI community had been eagerly waiting for was conspicuously absent. Engram, a memory-augmentation architecture jointly developed by DeepSeek and Peking University, is nowhere to be found in V4's final design.

The omission has sparked intense debate across AI research forums and social media. Many developers and researchers had assumed Engram would serve as the architectural bedrock of V4, and its absence has left a visible gap in what is otherwise one of the most technically ambitious open-weight model releases of 2025.

Key Takeaways

• Engram is missing from DeepSeek V4's technical report despite widespread expectations it would be included
• The technology was open-sourced in January 2025 by DeepSeek and Peking University, targeting memory efficiency in large language models
• Engram enables direct factual retrieval without activating the full deep network, saving both compute and VRAM
• Community members have called V4 'incomplete' without it, making it arguably the biggest regret of the release
• At least 3 follow-up papers have since expanded Engram's capabilities, suggesting the technology is far from dead
• DeepSeek has not publicly explained why Engram was excluded from the final V4 architecture

What Is Engram and Why Does It Matter?

Engram first appeared on arXiv in early January 2025, published as a joint research effort between DeepSeek and Peking University. At its core, Engram tackles one of the most persistent problems in large language model design: how models store and retrieve factual knowledge.

In traditional transformer architectures, even a simple factual query like 'What is the capital of the United Kingdom?' forces the model to propagate signals through its entire deep network. Every attention head, every feed-forward layer activates — all to retrieve a piece of information that could, in theory, be looked up in a fraction of the time.

Engram proposes an elegant alternative. It introduces a dedicated memory layer that stores frequently accessed factual knowledge in a retrievable format. When the model encounters a factual query, it can bypass the deep reasoning layers entirely and pull the answer directly from this memory store. The implications are significant:

• VRAM savings: Less activation memory needed for routine factual recall
• Freed capacity: Deep network layers can focus on higher-order reasoning tasks
• Faster inference: Direct retrieval is computationally cheaper than full forward passes
• Better scaling: Memory layers can scale independently from reasoning layers

This is not just an incremental optimization. It represents a fundamental rethinking of how knowledge and reasoning should be separated within a model's architecture.

Why Everyone Expected Engram in V4

The timing of Engram's release made its inclusion in V4 seem almost inevitable. Published just months before V4's announcement, the paper read like a preview of architectural decisions to come. The AI research community quickly connected the dots.

Discussions on platforms like Twitter/X, Reddit, and Chinese tech forums like Zhihu treated Engram as confirmed infrastructure for V4. Researchers analyzed its memory pooling mechanisms, debated its integration with Mixture of Experts (MoE) architectures, and speculated about how it might interact with DeepSeek's existing multi-head latent attention designs.

When V4's technical report finally dropped, the first instinct of countless readers was to hit Ctrl+F (or Command+F on Mac) and search for 'Engram.' The result: zero matches. The disappointment was immediate and vocal.

Some community members went as far as declaring V4 'architecturally incomplete' without Engram. While this may be hyperbolic — V4 is by all accounts a formidable model — the sentiment reveals just how much weight researchers placed on the technology.

What V4 Did Include Instead

To be fair, DeepSeek V4's technical innovations are substantial even without Engram. The model introduces several notable architectural and training advances:

• mHC (Multi-Head Clustering): A new approach to organizing attention heads for improved efficiency
• CSA (Cross-Sequence Attention): Enables attention mechanisms that span across sequence boundaries
• HCA (Hierarchical Context Aggregation): A layered approach to processing context at multiple granularities
• Muon Optimizer: A training optimization technique that reportedly improves convergence stability
• FP4 Quantization: Aggressive 4-bit floating-point quantization for reduced memory footprint during inference

Each of these innovations addresses real bottlenecks in LLM training and deployment. FP4 alone represents a significant step forward in making large models more accessible on consumer and enterprise hardware. The Muon optimizer, meanwhile, targets training efficiency — a critical concern as model sizes continue to grow.

Yet none of these directly address the knowledge-retrieval problem that Engram was designed to solve. They optimize how the model computes, but not how it remembers.

Engram Is Not Dead — 3 Follow-Up Papers Show Continued Development

Despite its absence from V4, Engram has not disappeared. In the weeks following V4's release, at least 3 notable follow-up papers have emerged, each extending Engram's capabilities in different directions.

The first explores a CXL memory pooling implementation. Compute Express Link (CXL) is a high-speed interconnect standard gaining traction in data center hardware. This paper places Engram's memory layers into a shared CXL memory pool accessible across multiple machines, directly addressing one of the biggest challenges in multi-node LLM deployment: how to efficiently share knowledge stores across distributed inference setups.

The second paper investigates conflict-free hot layer experiments. This work focuses on resolving contention issues when multiple inference requests simultaneously access the most frequently used portions of Engram's memory store. In high-throughput production environments, this kind of optimization is essential for maintaining low latency.

A third line of research — details of which are still emerging — appears to explore deeper integration patterns between Engram-style memory and reasoning architectures. Together, these papers suggest that Engram is being actively developed for future deployment, possibly in a subsequent model release.

The Broader Industry Context: Memory vs. Reasoning

Engram's story fits into a larger trend reshaping the AI industry. As models grow more capable, researchers are increasingly questioning whether monolithic architectures — where a single network handles everything from factual recall to complex reasoning — are the right approach.

Google DeepMind has explored similar territory with retrieval-augmented generation (RAG) enhancements. Meta's Llama team has experimented with external memory mechanisms. Anthropic has published research on how knowledge is encoded within transformer layers, hinting at potential future architectures that separate memory from computation.

The fundamental insight driving all of this work is the same one behind Engram: not every query requires the same computational depth. A model that can distinguish between 'I need to look something up' and 'I need to reason through this' will inherently be more efficient than one that treats every input identically.

This separation of concerns mirrors patterns in traditional computer architecture, where CPUs use cache hierarchies to avoid fetching every piece of data from main memory. Engram essentially proposes a 'knowledge cache' for LLMs.

What This Means for Developers and Businesses

For teams deploying large language models in production, the Engram question has practical implications. Current architectures force a trade-off: either deploy a massive model that can handle both factual and reasoning queries, or build complex RAG pipelines to offload factual retrieval to external databases.

Engram offers a potential middle path — a model that handles factual queries efficiently at the architecture level, without external retrieval infrastructure. For businesses, this could mean:

• Lower inference costs through reduced compute per factual query
• Simpler deployment without the need for separate vector databases and retrieval pipelines
• More predictable latency since factual queries bypass deep computation
• Better resource allocation as reasoning capacity is preserved for complex tasks

The fact that Engram did not make it into V4 means these benefits remain theoretical for now. But the continued research activity suggests they may materialize in a future release.

Looking Ahead: Will Engram Define DeepSeek V5?

The most likely explanation for Engram's absence from V4 is timing. Integrating a novel memory architecture into a production-scale model is not trivial. The CXL memory pooling and conflict-free hot layer papers suggest that key engineering challenges remain unsolved.

DeepSeek may have made a pragmatic decision: ship V4 with proven innovations like FP4 and Muon, while continuing to mature Engram for a future release. If this is the case, V5 — or whatever the next major release is called — could be the model where Engram finally finds its home.

The community will certainly be watching. Engram's absence from V4 has, paradoxically, only increased anticipation for its eventual deployment. When it does arrive in a production model, it could represent one of the most significant architectural shifts in LLM design since the introduction of Mixture of Experts.

For now, DeepSeek V4 stands as an impressive but — in the eyes of many researchers — incomplete step forward. The model's biggest regret may ultimately become its successor's biggest advantage.