ADE Adaptive Dictionary Embeddings: Breaking Through the Word Representation Bottleneck in Large Language Models

The Single-Vector Embedding Bottleneck Demands a Breakthrough

Word embedding is a cornerstone technology in natural language processing. From Word2Vec to the GPT series, virtually all language models rely on mapping words into representations within a continuous vector space. However, a long-standing fundamental problem has never been effectively resolved: traditional methods represent each word with a single vector, severely constraining the ability to capture the semantics of polysemous words.

Take the English word "bank" as an example — it carries entirely different meanings in "river bank," "bank account," and "bank on someone," yet in traditional word embeddings, they all share the same vector representation. This "one word, one vector" design creates a representation bottleneck, forcing models to compromise by averaging across different semantic meanings.

A recent paper published on arXiv introduces a new method called ADE (Adaptive Dictionary Embeddings), which for the first time successfully extends Multi-Anchor Representations to the scale of large language models, delivering a breakthrough for this foundational challenge.

ADE's Core Idea: A Paradigm Shift From Single-Anchor to Multi-Anchor

Multi-anchor representation means abandoning the use of a single vector to represent a word, instead using a combination of multiple vectors to capture the word's multiple semantic meanings. Previously, this approach had demonstrated significant advantages in small-scale models, but faced two core obstacles in practical application:

• Low computational efficiency: The parameter count and computational overhead introduced by multi-vector representations grow dramatically with model scale, becoming prohibitive
• Architectural integration difficulties: There was no effective scheme for integrating with modern large language model architectures such as Transformers

ADE's innovation lies in introducing an "adaptive dictionary" mechanism. Unlike traditional fixed multi-anchor allocation, ADE uses a learnable dictionary structure that allows the model to dynamically select and combine the most relevant anchor vectors based on context. This design preserves the rich semantic expressiveness of multi-anchor representations while drastically reducing computational costs through an adaptive sparse activation strategy, making practical deployment on large-scale models feasible.

Technical Significance