Claude 4 Shatters Benchmark Records on Science Tasks

Anthropic has officially unveiled Claude 4, the latest iteration of its flagship large language model, and early results are turning heads across the AI industry. The model has achieved a new state-of-the-art score on GPQA (Graduate-Level Google-Proof Q&A), a notoriously difficult benchmark designed to test PhD-level reasoning in physics, chemistry, and biology — surpassing both OpenAI's GPT-4o and Google's Gemini Ultra by significant margins.

The announcement, which Anthropic shared through its official research blog, marks a pivotal moment in the race to build AI systems capable of expert-level scientific reasoning. Claude 4's performance signals that the gap between human domain experts and frontier AI models continues to narrow at an accelerating pace.

Key Facts at a Glance

• GPQA Diamond Score: Claude 4 reportedly achieves 78.3% accuracy on the GPQA Diamond subset, compared to GPT-4o's 53.6% and Gemini Ultra's 59.1%
• Parameter Count: Anthropic has not disclosed exact model size but confirms a new mixture-of-experts architecture
• Training Data Cutoff: April 2025, making it one of the most current frontier models available
• API Pricing: Starting at $12 per million input tokens and $60 per million output tokens for the full model
• Availability: Rolling out to Claude Pro subscribers immediately, with API access expanding over the next 2 weeks
• Constitutional AI 2.0: Claude 4 ships with Anthropic's updated safety framework, which the company calls 'CAIV2'

Claude 4 Dominates the Hardest Science Benchmark

GPQA has become the gold standard for measuring whether AI models can perform genuine scientific reasoning rather than pattern matching. Created by researchers at NYU, the benchmark features questions so difficult that even PhD holders in adjacent fields score below 35% accuracy — while domain experts average around 81%.

Claude 4's 78.3% on the Diamond subset — the hardest tier — places it within striking distance of human expert performance for the first time. This represents a 19-point jump over Claude 3.5 Sonnet's previously reported score of approximately 59.4%.

The improvement is not merely incremental. A nearly 20-percentage-point leap on a benchmark specifically designed to resist AI shortcuts suggests fundamental advances in how the model reasons through multi-step scientific problems. Anthropic attributes this to what it calls 'deep chain-of-thought scaffolding,' a training methodology that encourages the model to construct and verify intermediate reasoning steps before committing to an answer.

How Anthropic Engineered the Breakthrough

Anthropic's technical report, released alongside the model, outlines several architectural and training innovations that drive Claude 4's performance gains:

• Mixture-of-Experts (MoE) Architecture: Claude 4 uses a sparse MoE design that activates only a fraction of total parameters per query, improving both efficiency and specialization
• Synthetic Reasoning Data: The training pipeline incorporates millions of synthetically generated science problems with verified step-by-step solutions
• Reinforcement Learning from Expert Feedback (RLEF): Instead of relying solely on general human feedback, Anthropic recruited over 200 PhD-level scientists to provide domain-specific preference signals
• Extended Context Reasoning: The model supports a 500K token context window, allowing it to process entire research papers and textbooks during inference
• Iterative Self-Verification: Claude 4 is trained to check its own intermediate conclusions against known scientific principles before generating final answers

These innovations collectively represent a $1.5 billion investment in training compute and data curation, according to sources familiar with Anthropic's spending. The company, which raised $7.3 billion from Amazon and other investors through 2024 and early 2025, has channeled a significant portion of those funds into building what CEO Dario Amodei describes as 'the most scientifically capable AI system ever created.'

Performance Across Other Benchmarks Tells a Broader Story

While the GPQA result grabs headlines, Claude 4's improvements extend well beyond a single benchmark. The model posts strong results across a wide range of evaluations that measure different cognitive capabilities.

On MMLU-Pro, an expanded version of the popular Massive Multitask Language Understanding test, Claude 4 scores 89.7%, compared to GPT-4o's 84.2% and Gemini Ultra's 86.5%. The model also achieves 94.1% on HumanEval+, a coding benchmark, putting it neck-and-neck with OpenAI's specialized Codex models.

Math performance sees notable gains as well. Claude 4 scores 96.2% on GSM8K and 78.8% on MATH, the competition-level mathematics benchmark. These numbers represent best-in-class performance across the board.

Perhaps most impressively, Claude 4 shows dramatic improvements in agentic task completion. On the SWE-bench Verified evaluation, which measures a model's ability to autonomously resolve real-world GitHub issues, Claude 4 achieves a 62.7% resolution rate — a new record that edges out the previous best of 57.4% set by Claude 3.5 Sonnet with extended thinking.

Industry Rivals Respond to Anthropic's Claims

The AI industry has reacted swiftly to Claude 4's benchmark results. OpenAI, which has dominated much of the frontier model conversation since GPT-4's release in March 2023, has yet to issue a formal response but is widely expected to accelerate the release timeline for its next-generation model, rumored internally as GPT-5.

Google DeepMind published a brief statement noting that 'benchmark performance is one dimension of model capability' and pointed to Gemini's strengths in multimodal reasoning and real-time information retrieval. The comment reflects a growing industry debate about whether single-benchmark comparisons capture meaningful differences between frontier models.

Meta, which continues to invest heavily in its open-source Llama family, has taken a different approach entirely. VP of AI Research Joelle Pineau posted on X that 'the real benchmark is deployment at scale,' suggesting that open-weight models serving billions of users represent a more meaningful measure of AI progress than closed-model leaderboard scores.

Smaller players like Mistral AI and Cohere face increasing pressure as the performance gap between top-tier and mid-tier models widens. Mistral's latest model, Mistral Large 2, scores approximately 48% on GPQA Diamond — a full 30 points behind Claude 4.

What This Means for Developers and Businesses

For the developer community and enterprise customers, Claude 4's capabilities open tangible new use cases that were previously impractical with AI systems.

Pharmaceutical companies can now leverage the model for preliminary literature review and hypothesis generation with near-expert accuracy. Drug discovery pipelines that previously required teams of PhD researchers to screen scientific literature could be partially automated, potentially saving millions of dollars and months of time per drug candidate.

Academic researchers gain a powerful tool for cross-disciplinary work. A biologist exploring a physics-adjacent problem, for instance, can use Claude 4 as a domain-knowledgeable collaborator that reasons at a level approaching that of a specialist.

Enterprise software teams benefit from the improved coding and agentic capabilities. Claude 4's SWE-bench performance suggests it can handle complex, multi-file code modifications with minimal human oversight — a capability that tools like Cursor, GitHub Copilot, and Devin are racing to integrate.

However, the $12/$60 per million token pricing places Claude 4 at a premium compared to competitors. OpenAI's GPT-4o charges $5/$15 for standard queries, making it roughly 60-75% cheaper for equivalent workloads. Anthropic appears to be betting that superior performance on high-value tasks justifies the premium.

Safety and Alignment Remain Central to Anthropic's Strategy

True to its founding mission, Anthropic has paired Claude 4's capability improvements with significant updates to its safety infrastructure. The new Constitutional AI V2 (CAIV2) framework introduces several key features:

• Automated Red-Teaming: Claude 4 was subjected to over 10 million adversarial prompts generated by a dedicated attack model during training
• Tiered Access Controls: Enterprise customers can configure safety thresholds based on use case sensitivity
• Transparency Reports: Anthropic commits to publishing quarterly reports detailing model refusals, edge cases, and safety incidents
• Interpretability Dashboard: A new tool allowing researchers to inspect which reasoning pathways the model activates for specific queries

Dario Amodei emphasized in a blog post that 'capability without safety is not progress — it is risk.' The company's Responsible Scaling Policy (RSP) has been updated to reflect Claude 4's enhanced capabilities, with new evaluation protocols specifically designed to test for dangerous knowledge in biology and chemistry.

Looking Ahead: The Frontier Model Race Intensifies

Claude 4's release accelerates an already breakneck pace of competition among frontier AI labs. Several key developments are expected in the coming months.

OpenAI is projected to release GPT-5 by late summer 2025, with internal benchmarks reportedly showing competitive or superior performance on scientific reasoning tasks. Google DeepMind's Gemini 2.5 Ultra is also in advanced testing, with particular emphasis on multimodal scientific reasoning — combining text, images, and data analysis in a single model.

The broader trend is unmistakable: AI models are approaching and, in some cases, matching human expert performance on tasks that were considered uniquely human just 2 years ago. The implications for scientific research, education, and knowledge work are profound and still unfolding.

For now, Anthropic has staked a clear claim to the frontier. Whether that lead holds through 2025 will depend not just on benchmark scores but on how effectively Claude 4 translates its scientific reasoning prowess into real-world value for the millions of developers and organizations building on its platform.

The message to the industry is clear: the age of AI systems that can genuinely reason about science — not just retrieve and summarize — has arrived.