AI Evaluation Is Becoming the New Computing Power Bottleneck

When Evaluation Can't Keep Up with Model Evolution

Over the past two years, the core narrative in the AI industry has revolved around "compute" — whoever commands more GPU clusters can train more powerful models. Yet a new bottleneck is quietly emerging: AI evaluation (Evals) itself is becoming a critical obstacle to model iteration and deployment.

From OpenAI and Anthropic to Google DeepMind, virtually every leading AI lab faces the same dilemma: model capabilities are advancing far faster than evaluation systems can evolve. When we cannot accurately measure where a model excels or by how much it has improved, the entire R&D feedback loop breaks down. Evaluation — once regarded as "auxiliary work" — is becoming the new computing power bottleneck of our era.

Three Dimensions of the Evaluation Crisis

Dimension One: The "Ceiling Effect" of Benchmarks

Traditional benchmarks are being "maxed out" at an unprecedented pace. Evaluation sets once considered highly challenging — MMLU, HumanEval, GSM8K — have now been nearly or fully solved by multiple models. When every top model scores above 95 on the same leaderboard, the benchmark loses its discriminative power.

The industry is forced to continually develop new, harder benchmarks — from MMLU to MMLU-Pro, from HumanEval to SWE-bench, from GSM8K to MATH to FrontierMath. But designing, validating, and promoting each new benchmark demands significant time and resources, while model capabilities often render a new benchmark "obsolete" again within months.

The fundamental problem with this cat-and-mouse game is: we can no longer build evaluations faster than models can conquer them.

Dimension Two: Soaring Compute Costs of Evaluation Itself

Evaluation is no longer as simple as "running a few test cases." Modern AI evaluation has become extraordinarily expensive:

• Multi-turn dialogue evaluation requires models to engage in extended contextual interaction, with a single evaluation potentially consuming thousands of API calls
• Code capability evaluation (e.g., SWE-bench) demands that models modify and debug real code repositories, with each test case potentially requiring tens of minutes of inference time
• Agent evaluation (Agent Evals) requires continuous model-environment interaction, with a complete evaluation potentially taking hours or even days
• Safety evaluation (Red Teaming) demands large-scale adversarial testing, often requiring one powerful model to evaluate another, doubling compute consumption

Take Anthropic's model safety evaluation as an example: before releasing a new version, its internal evaluation pipeline may consume compute equivalent to a small-scale training run. For scenarios requiring "LLM-as-Judge" approaches, evaluation costs can rival those of inference services themselves.

When evaluation itself demands massive compute, it is no longer a "lightweight" verification step but the third major compute consumption scenario alongside training and inference.

Dimension Three: The Unscalability of Human Evaluation

Across many critical capability dimensions, automated evaluation still cannot replace human judgment. The quality of creative writing, the naturalness of conversation, the reliability of reasoning processes, the helpfulness of responses — all of these depend heavily on the judgment of expert human evaluators.

But human evaluation faces severe scalability challenges:

• Skilled evaluators are scarce and expensive, especially when domain experts (in medicine, law, mathematics) are needed
• Consistency in human evaluation is difficult to guarantee, with significant judgment variance across evaluators
• Evaluation speed cannot keep pace with model iteration cycles; a large-scale human evaluation may take weeks
• As model capabilities approach or surpass average human levels, finding people "qualified" to judge models becomes increasingly difficult

This creates a paradox: the more powerful the model, the more difficult, expensive, and time-consuming it becomes to evaluate.

Chain Reactions: How the Evaluation Bottleneck Affects the Entire AI Ecosystem

The impact of the evaluation bottleneck extends far beyond the technical level, profoundly affecting the entire AI value chain.

Impact on R&D efficiency: When research teams cannot quickly and accurately assess the effects of model improvements, R&D iteration slows down. A training experiment might be completed in days, but comprehensively evaluating its results may take even longer. This means teams are often "flying blind" — making changes without certainty that genuine improvements have been achieved.

Impact on business decisions: Enterprise customers rely heavily on evaluation results when choosing AI models. When existing evaluations fail to accurately reflect model performance in real business scenarios, procurement decisions become difficult. This is why an increasing number of enterprises are building their own proprietary evaluation systems, further adding to the industry's overall costs.

Impact on safety governance: Regulators and safety researchers need evaluations to determine whether models are safe and controllable. If evaluation systems are insufficiently robust or timely, two extremes may emerge: either excessive conservatism that stifles innovation, or insufficient assessment that allows risky models to be released prematurely.

Impact on academic research: When top models' performance on public benchmarks approaches saturation, the academic paper "score-chasing" paradigm breaks down. Researchers must invest more effort in designing new evaluation methods, which itself is a resource-intensive research endeavor.

How the Industry Is Responding

Facing the evaluation bottleneck, the industry is seeking breakthroughs from multiple directions:

1. Dynamic Evaluation and Private Benchmarks

Some organizations are adopting dynamic evaluation strategies, continuously updating test sets to prevent data contamination and overfitting. The "crowdsourced blind evaluation" model used by Chatbot Arena is an innovative approach, assessing models through real users' real-time preference votes and avoiding the ceiling problem of static benchmarks. Companies like Scale AI are also building enterprise-grade private evaluation platforms to help clients establish customized evaluation systems for specific scenarios.

2. Tiered Evaluation Architecture

The industry is exploring a tiered evaluation architecture of "rapid rough screening + deep precision assessment." Lightweight automated evaluations are used for rapid screening during routine iterations, with expensive deep assessments reserved only for model versions that pass the initial screening. This approach can significantly reduce overall costs while maintaining evaluation quality.

3. Evolution of LLM-as-Judge

The "LLM-as-Judge" paradigm — using strong models to evaluate weaker ones — is maturing rapidly. While this approach also consumes compute, its scalability far exceeds that of human evaluation. Researchers are developing more reliable evaluation protocols, including multi-model cross-evaluation, structured scoring frameworks, and other methods to improve the accuracy and consistency of automated assessment.

4. Professionalization of Evaluation Infrastructure

Evaluation is evolving from "auxiliary work" into an independent infrastructure domain. Specialized evaluation companies and platforms are emerging, offering standardized evaluation services and tools. Some cloud computing providers are also beginning to offer evaluation capabilities as part of their AI infrastructure.

5. Evaluation Design Targeting Capability Boundaries

Rather than pursuing "comprehensive evaluation," some researchers are focusing on probing models' capability boundaries — identifying the conditions under which models fail. This "stress test" approach to evaluation is more efficient and better reveals a model's true level. FrontierMath, launched by Epoch AI, embodies this philosophy, focusing on testing the limits of mathematical reasoning capabilities.

A Deeper Question: What Are We Actually Evaluating?

Behind the evaluation bottleneck lies a more fundamental question: our definition and measurement of "intelligence" may have been flawed from the very beginning.

Current evaluation systems are mostly based on a "task completion" paradigm — giving a model a question and checking whether it can provide the correct answer. But as AI capabilities advance, what we truly care about may not be "how many questions were answered correctly," but rather the reliability of reasoning, the accurate boundaries of knowledge, alignment with human values, and adaptability in open-ended scenarios.

Mature methodologies for evaluating these dimensions do not yet exist. We are in an awkward transitional period: the old evaluation paradigm is failing, while the new one has yet to be established.

Looking Ahead: Evaluation Will Become AI's New Competitive Battleground

Looking to the future, evaluation will likely become the next critical battleground in AI competition. Just as the compute race gave rise to NVIDIA's dominance, the evaluation race may produce new winners.

Foreseeable trends include:

• Dedicated compute investment in evaluation will increase dramatically, with leading labs potentially allocating 10%–20% or even higher shares of their compute budgets to evaluation
• Evaluation methodology will become a core competitive advantage — whoever can more accurately assess model capabilities will operate more efficiently