GAIA-v2-LILT: A Multilingual AI Agent Benchmark That Goes Beyond Translation

The English-Dominated AI Agent Evaluation Dilemma

The field of AI agents is advancing rapidly, yet its evaluation benchmarks face a long-overlooked problem — nearly all mainstream benchmarks are English-centric. When researchers attempt to extend these benchmarks to other languages, they typically rely solely on machine translation (MT) with limited human post-editing. This "minimal workflow" appears efficient but conceals significant pitfalls.

A recent paper published on arXiv (arXiv:2604.24929v1) formally introduces GAIA-v2-LILT (Linguistically and Interculturally Localized Tasks), providing an entirely new methodological framework for building multilingual agent benchmarks that directly addresses the core deficiencies of translation-based benchmarks.

Why Machine Translation Falls Short for Agent Tasks

Unlike traditional NLP tasks, agent benchmarks involve complex multi-step reasoning, tool invocation, and real-world interaction. The researchers point out that simple machine translation in such tasks easily leads to two critical problems:

First, query-answer misalignment. Agent tasks often feature complex logical chains between questions and answers. Subtle deviations during translation can cause questions to point to incorrect answers, or render answers invalid in the target language context. For example, a task requiring users to query information from a specific English-language webpage may become inaccessible to target-language users after direct translation.

Second, culturally off-target context. Many agent tasks are embedded in specific cultural backgrounds, such as legal and regulatory queries, local service recommendations, and currency conversions. Mechanical translation cannot handle these deep cultural contexts, causing benchmarks to lose practical relevance in the target language environment.

The Core Methodology of GAIA-v2-LILT: Functional Alignment and Cultural Adaptation

To address these issues, the research team proposed a refined benchmark adaptation workflow. Its core philosophy shifts from "word-for-word translation" to "functional equivalence." The workflow comprises several key components:

Functional Alignment: Ensuring that translated tasks maintain functional equivalence with the original tasks in the target language. This means not only translating surface-level text but also confirming that the tools, data sources, and reasoning paths involved in the task remain valid and accessible in the target language environment.

Cultural Adaptation: Localizing tasks involving culture-specific content by replacing original scenarios with equivalent scenarios from the target culture, rather than simply translating culturally unique concepts. For instance, tasks involving the U.S. tax system would be replaced with corresponding tax scenarios from the target country.

Explicit Verification Mechanisms: After adaptation, structured verification steps ensure consistency among each task's question, reasoning path, and expected answer, fundamentally preventing implicit errors introduced through translation.

Far-Reaching Implications for Multilingual AI Evaluation

The significance of this work extends well beyond improving a single benchmark. As AI agent applications rapidly deploy worldwide, accurately assessing model performance in non-English environments has become an urgent industry need.

Currently, mainstream agent benchmarks including GAIA and WebArena primarily serve English-language scenarios. This means a large number of model developers in non-English regions lack reliable evaluation tools, potentially leading to discrepancies between evaluation results and real-world deployment performance. The methodology proposed by GAIA-v2-LILT provides a reusable paradigm for building truly reliable multilingual agent evaluation systems.

Furthermore, this research reminds the industry that the principle "translation does not equal localization" applies equally to the domain of technical evaluation in the era of AI globalization. Multilingual benchmarks built solely through machine translation not only fail to accurately reflect model capabilities but may also mislead research and development priorities.

Outlook: Toward Truly Globalized Agent Evaluation

As the multilingual capabilities of large language models continue to strengthen and agent applications accelerate their penetration into global markets, establishing high-quality multilingual evaluation standards has become imperative. The introduction of GAIA-v2-LILT marks a turning point where the community begins to confront the limitations of "translation-based benchmarks" and explore more rigorous solutions.

Looking ahead, we can expect more benchmarks to incorporate multilingual and multicultural considerations from the design stage, rather than treating them as appendages to English versions. This will not only promote fair evaluation of AI agent technology on a global scale but also foster more inclusive AI technology development.