IIT Bombay Builds Benchmark for Low-Resource AI

IIT Bombay has unveiled a comprehensive benchmark framework designed to rigorously evaluate large language model performance across low-resource languages — those with limited training data and digital presence. The initiative addresses a critical blind spot in AI development, where models like GPT-4, Claude, and Gemini excel in English but often falter dramatically when handling languages spoken by billions of people worldwide.

The benchmark, developed by researchers at IIT Bombay's Department of Computer Science and Engineering, covers multiple evaluation dimensions including text generation, comprehension, translation, and reasoning across more than 30 languages. It represents one of the most systematic efforts to date to hold AI systems accountable for their multilingual capabilities — or lack thereof.

Key Facts at a Glance

• Scope: The benchmark evaluates AI models across 30+ low-resource languages, including several Indian, African, and Southeast Asian languages
• Tasks covered: Text generation, reading comprehension, machine translation, named entity recognition, sentiment analysis, and logical reasoning
• Models tested: Preliminary evaluations include GPT-4, Llama 3, Gemma, and several multilingual models like BLOOM and IndicBERT
• Performance gap: Leading commercial models score up to 40-60% lower on low-resource language tasks compared to their English benchmarks
• Open access: The benchmark dataset and evaluation toolkit are released under an open-source license for community use
• Collaboration: The project involves partnerships with linguists and native speakers from over 15 countries

Why Current Benchmarks Fall Short

Most widely used AI benchmarks — from MMLU to HumanEval to HellaSwag — are overwhelmingly English-centric. Even ostensibly multilingual benchmarks like XTREME and MEGA tend to focus on a narrow set of high-resource languages such as French, German, Chinese, and Spanish.

This leaves a massive evaluation gap. Languages like Marathi, Yoruba, Khmer, Odia, and Tigrinya — collectively spoken by hundreds of millions of people — lack standardized testing frameworks. Without proper benchmarks, model developers have no reliable way to measure progress or identify failures in these languages.

IIT Bombay's framework directly confronts this problem. Unlike previous benchmarks that simply translate English test sets (often introducing cultural and contextual errors), the new benchmark constructs evaluation tasks natively in each target language. Native speakers and linguists collaborate to create culturally appropriate test cases that reflect genuine language use rather than awkward translations.

Inside the Benchmark Architecture

The framework is organized into 6 core evaluation pillars, each targeting a distinct capability:

• Text Generation Quality: Measures fluency, coherence, and grammatical accuracy in open-ended generation tasks
• Reading Comprehension: Tests whether models can extract and synthesize information from passages written in the target language
• Machine Translation: Evaluates bidirectional translation between each low-resource language and English, as well as between language pairs within the same family
• Named Entity Recognition (NER): Assesses the ability to identify people, places, organizations, and dates in context
• Sentiment and Tone Analysis: Checks whether models can detect emotional valence and rhetorical intent across different cultural contexts
• Logical Reasoning: Presents chain-of-thought and multi-step reasoning problems constructed natively in each language

Each pillar includes both automated metrics (BLEU, ROUGE, F1 scores) and human evaluation protocols. The researchers emphasize that automated metrics alone are insufficient for low-resource languages because reference corpora are often too small or noisy to serve as reliable baselines.

A Focus on Linguistic Diversity

The benchmark deliberately includes languages from diverse typological families. Agglutinative languages like Tamil and Turkish, tonal languages like Yoruba and Vietnamese, and morphologically rich languages like Hindi and Swahili all present unique challenges that expose different failure modes in current LLMs.

This typological diversity is intentional. The researchers argue that a model's ability to handle English — an analytic language with relatively simple morphology — tells us almost nothing about its capability with structurally different languages.

Early Results Reveal Stark Performance Gaps

Preliminary evaluations using the benchmark paint a sobering picture. GPT-4, widely regarded as the most capable commercial LLM, scores approximately 85-92% on English-language comprehension and reasoning tasks. On equivalent tasks in languages like Odia, Assamese, and Yoruba, its scores drop to the 35-55% range.

Meta's Llama 3 (70B parameter version) shows a similar pattern, though its open-weight nature has allowed community fine-tuning that narrows the gap slightly for some Indian languages. Google's Gemma models perform comparably on South and Southeast Asian languages, likely benefiting from Google's multilingual training data pipelines.

Interestingly, smaller specialized models sometimes outperform larger general-purpose ones. IndicBERT and models from the AI4Bharat initiative — also based at IIT — score competitively on Indian language tasks despite having a fraction of the parameters. This suggests that targeted training data and architectural choices can partially compensate for raw scale.

The most dramatic failures appear in generation tasks. Models frequently produce grammatically broken output, mix scripts inappropriately, or default to English vocabulary when they lack confidence in the target language. Reasoning tasks in low-resource languages also expose a tendency for models to 'hallucinate' more frequently than they do in English.

Industry Context: A Growing Push for Multilingual AI

IIT Bombay's benchmark arrives at a pivotal moment. Major tech companies are investing heavily in multilingual AI capabilities, recognizing that the next billion internet users will not be English speakers.

Google has poured resources into its 1,000-language initiative, aiming to build AI models that support the world's most-spoken languages. Meta released the No Language Left Behind (NLLB) translation model covering 200 languages. Microsoft has expanded its Project ELLORA to improve AI accessibility for Indian languages.

Yet despite these efforts, progress has been uneven. Corporate benchmarks tend to showcase favorable results, and there is no independent, comprehensive standard for measuring multilingual AI quality. IIT Bombay's framework fills this gap by providing a neutral, academically rigorous evaluation platform.

Compared to existing multilingual benchmarks like XTREME-R (which covers around 50 languages but focuses on classification tasks) or MEGA (which evaluates generative capabilities but in a narrower set), IIT Bombay's offering is notable for its breadth of tasks, its insistence on native-language test construction, and its inclusion of human evaluation protocols.

What This Means for Developers and Businesses

For AI developers building products for multilingual markets, the benchmark provides an essential reality check. Companies deploying chatbots, content moderation systems, or translation tools in regions like South Asia, Sub-Saharan Africa, or Southeast Asia can now measure their models against a standardized yardstick.

The practical implications are significant:

• Product teams can identify specific language-task combinations where their models underperform before deploying to production
• Researchers gain a shared evaluation framework that enables apples-to-apples comparison across different modeling approaches
• Policymakers can reference benchmark results when setting standards for AI deployment in government services and education
• Investors evaluating multilingual AI startups now have an independent performance metric beyond self-reported accuracy claims

The open-source release is particularly valuable. Startups and academic labs with limited budgets can access the full evaluation toolkit without licensing fees, lowering the barrier to rigorous multilingual AI development.

Looking Ahead: Expanding Coverage and Community Adoption

The IIT Bombay team has outlined an ambitious roadmap. Over the next 12-18 months, they plan to expand the benchmark to cover 50+ languages, with particular emphasis on African languages and indigenous languages of the Americas — regions that remain almost entirely absent from AI evaluation frameworks.

The researchers are also developing a leaderboard platform where model developers can submit results and track progress over time, similar to how the Open LLM Leaderboard on Hugging Face catalyzed competition and transparency in English-language model development.

Community contributions will be critical. The team has issued an open call for native speakers, linguists, and regional AI researchers to help construct evaluation datasets for underrepresented languages. This crowdsourced approach mirrors the methodology that made benchmarks like BIG-Bench successful — scaling through distributed expertise rather than centralized effort.

If widely adopted, IIT Bombay's benchmark could fundamentally shift how the AI industry thinks about model quality. Rather than treating multilingual capability as a secondary feature, developers may increasingly be expected to demonstrate robust low-resource language performance as a baseline requirement. In a world where only about 20% of the global population speaks English, that shift is long overdue.