TOON Format: Dramatically Reducing Token Overhead in LLM Prompts

Introduction: JSON's Hidden Costs Are Devouring Your Token Budget

If you're building RAG pipelines, Agent tool chains, or frequently calling LLM APIs, you've likely encountered a frustrating reality — in a single prompt, the truly valuable data itself doesn't consume many tokens. What actually eats into your budget are the JSON structural characters that repeatedly surround the data. Curly braces, square brackets, repeated key names, redundant quotation marks… these structural overheads multiply exponentially in batch data scenarios.

A new encoding format called TOON (Token-Oriented Object Notation) is attempting to solve this problem at its root.

Core Philosophy: Schema-Once, Data-Many

TOON's design philosophy can be summed up in one sentence: declare the schema once, transmit data many times.

In traditional JSON format, every record must carry complete field names and structural markers. Suppose you need to embed 100 user records in a prompt, each containing field names like "name," "age," and "email" — these key names get written out 100 times. For human reading, this is redundant but acceptable. For LLM calls billed by token, however, this is real cost waste.

TOON's approach completely separates structure definitions (Schema) from data content:

• Schema Layer: Declare the field structure once at the beginning of the data block, including field names, types, and hierarchical relationships
• Data Layer: Each subsequent record only needs to arrange values in the declared order, without repeating key names

This design preserves JSON's complete logical data model — objects, arrays, and nested structures are all intact — but compresses repeated information to a minimum at the physical encoding level.

Technical Analysis: Why LLM Scenarios Especially Need This Optimization

Quantitative Comparison of Token Overhead

Take a typical RAG retrieval scenario as an example: suppose each retrieval returns 20 document fragments, each carrying four fields — "title," "content," "source," and "score." In standard JSON, field names and structural characters alone consume approximately 60 tokens per record. Across 20 records, that's roughly 1,200 tokens used purely for structural expression. Under the TOON format, the schema declaration requires only about 30 tokens, and the structural overhead per subsequent record drops to fewer than 10 tokens, totaling approximately 230 tokens — a savings of nearly 80%.

Compatibility with LLM Comprehension

A key design consideration of TOON is human readability. Unlike binary compression schemes such as Protobuf and MessagePack, TOON remains a plain text format that LLMs can directly understand and parse. This means developers don't need to include extra explanations of the data format in their prompts — models can correctly extract information through their inherent ability to understand structured text.

Application Potential in Agents and Tool Chains

In multi-step Agent workflows, passing intermediate states often involves extensive serialization and deserialization of structured data. The TOON format can effectively reduce context window usage, allowing Agents to carry more useful information within limited context lengths. This advantage is particularly pronounced in scenarios involving tabular data, database query results, or batch API responses.

Potential Challenges and Limitations

Although TOON's concept is highly appealing, several challenges remain for real-world adoption:

• Ecosystem Compatibility: JSON enjoys native support across virtually all programming languages and a massive tool ecosystem. TOON needs to build its own parsers and conversion tool chains.
• Model Adaptation Differences: Different LLMs vary in their ability to understand non-standard formats. TOON's performance may differ across models like GPT-4o, Claude, and Qwen, requiring extensive empirical testing.
• Debugging Readability: While TOON maintains human readability, the separation of Schema and Data may impose additional cognitive load on developers when debugging long prompts to match fields.
• Dynamic Structure Scenarios: For data with unfixed fields and highly dynamic structures, TOON's "Schema-Once" advantage is diminished.

Industry Context: Token Efficiency Optimization Has Become Essential

TOON's emergence is not an isolated case. As LLM applications move from experimentation to production, token cost optimization is becoming a core concern in engineering practice. From OpenAI's "Predicted Outputs" feature to various prompt compression tools (such as LLMLingua) and the proliferation of Context Caching technologies, the industry is exploring paths to reduce token consumption from multiple dimensions.

TOON takes the approach of addressing the fundamental layer of data encoding formats, offering a universal optimization solution that doesn't depend on model providers and doesn't alter business logic — an approach worth paying attention to.

Outlook: The LLM-Native Era of Data Formats

JSON was born in the Web 2.0 era, designed for data exchange that serves both humans and machines. Under the new paradigm of LLM-native applications, data format design needs to revisit the fundamental question of "who are we optimizing for?" TOON represents a new direction of thinking: data encoding born for token economics.

In the future, we may see more data formats and protocols custom-built for LLM scenarios. When every single token impacts cost and performance, data expression efficiency will no longer be an engineering detail that can be ignored — it will become a first-class citizen in architecture design.

For teams using RAG or Agents at scale, keeping an eye on the evolution of TOON and similar projects may be the next breakthrough in controlling costs and improving efficiency.