MATH-PT: The First Mathematical Reasoning Benchmark for Portuguese Unveiled

The Language Gap in Mathematical Reasoning Evaluation Demands Attention

Large language models (LLMs) are making rapid strides in complex mathematical reasoning, but a long-overlooked issue is drawing increasing attention from the academic community: virtually all mainstream mathematical reasoning benchmarks are built around English, leaving non-English languages severely underserved in evaluation resources. A recent study published on arXiv (arXiv:2604.25926v1) has officially introduced the MATH-PT dataset, specifically targeting European Portuguese and Brazilian Portuguese, marking a critical step toward filling this evaluation gap.

MATH-PT: Not Just Translation, But Native Construction

Unlike previous approaches that simply machine-translated English problems into other languages, MATH-PT's design philosophy emphasizes linguistic "nativeness." The research team noted that a major shortcoming of existing multilingual math benchmarks is that they are often literal translations of English datasets, which not only introduce translation noise but also fail to reflect the target language's authentic mathematical conventions and cultural nuances.

The core innovations of MATH-PT include:

• Dual-variant coverage: Encompassing both European Portuguese (PT-EU) and Brazilian Portuguese (PT-BR), with full consideration of differences in terminology, expression patterns, and educational systems between the two variants
• Multi-level difficulty design: Spanning multiple difficulty levels from basic arithmetic to advanced mathematics, comprehensively testing models' mathematical reasoning capabilities
• Linguistic authenticity: Problems designed to align with actual mathematics education practices in Portuguese-speaking countries, rather than being stiff translation artifacts

Revealing the True Weaknesses of LLM Multilingual Reasoning

The release of this study carries significant practical implications. Portuguese is the world's sixth most spoken language, with over 250 million native speakers across Europe, South America, Africa, and beyond. However, until now, there was no reliable evaluation tool to answer the question of how well mainstream LLMs actually perform in mathematical reasoning within Portuguese-language contexts.

The research team pointed out that relying solely on English benchmarks to assess a model's mathematical abilities can create a serious "capability illusion" — strong performance on English math problems does not necessarily mean the model is equally reliable in other languages. This evaluation bias poses a significant risk when deploying LLMs in educational, research, and commercial settings in Portuguese-speaking countries.

The Global Trend Toward Multilingual AI Evaluation

The emergence of MATH-PT is not an isolated case but part of a broader global wave of multilingual AI evaluation. In recent years, benchmarks such as C-Eval and CMATH for Chinese, JGLUE for Japanese, and ArabicMMLU for Arabic have successively appeared, collectively driving AI evaluation from an "English-centric" paradigm toward a "multilingual-balanced" approach.

However, exploration of multilingual mathematical reasoning evaluation remains in its early stages. Compared to natural language understanding and commonsense reasoning, mathematical reasoning demands greater linguistic precision — subtle wording differences in problems can lead to semantic shifts, making the construction of high-quality multilingual math benchmarks far more challenging than general tasks.

Looking Ahead: Toward Fairer AI Capability Assessment

The launch of MATH-PT sets a new standard for multilingual mathematical reasoning evaluation. As LLMs accelerate their penetration into global education, ensuring that models possess reliable mathematical reasoning capabilities in non-English environments is no longer a "nice-to-have" academic pursuit — it is a core issue of AI fairness.

In the future, the research team is expected to expand this work to additional languages while providing other language researchers with a reusable methodological framework. For LLM developers, MATH-PT offers a new "mirror" — helping them see the true mathematical reasoning capabilities of their models beyond the halo of English.