Sakana AI Breaks Ground With Evolutionary Model Merging

Sakana AI, the Tokyo-based startup backed by SoftBank and founded by former Google researchers, has achieved a significant breakthrough in evolutionary model merging — a technique that combines multiple pre-existing AI models into superior new ones without the massive computational costs of training from scratch. The approach could fundamentally reshape how the AI industry thinks about building large language models, offering a dramatically cheaper and faster alternative to conventional model development.

The research, which has drawn attention from leading AI labs across Silicon Valley and beyond, demonstrates that evolution-inspired algorithms can automatically discover optimal ways to merge the weights and architectures of different foundation models. The result is new models that outperform their parent models on targeted tasks — all while consuming a fraction of the compute resources typically required.

Key Takeaways

• Evolutionary model merging uses nature-inspired algorithms to combine existing AI models into new, more capable systems
• The technique eliminates the need for expensive GPU-intensive retraining, potentially saving millions of dollars per model
• Sakana AI has raised over $300 million in funding, with SoftBank leading its latest round at a reported $2 billion valuation
• The approach produced models that outperform individual source models on domain-specific benchmarks
• Co-founded by Llion Jones, a co-author of the landmark 'Attention Is All You Need' transformer paper, and David Ha, former head of research at Stability AI
• The method could democratize advanced AI development for companies without hyperscaler-level compute budgets

How Evolutionary Model Merging Actually Works

Traditional AI model development follows a predictable — and expensive — pattern. Companies like OpenAI, Google, and Anthropic spend tens of millions of dollars training massive neural networks on enormous datasets using thousands of GPUs over weeks or months. Sakana AI's approach sidesteps much of this process entirely.

Evolutionary model merging draws inspiration from biological evolution. Instead of training a single model from scratch, the technique starts with a population of existing open-source models — such as variants of Meta's Llama, Mistral's models, or specialized fine-tuned derivatives. An evolutionary algorithm then systematically explores different ways to combine these models' parameters, architectures, and learned representations.

The algorithm evaluates each 'offspring' model against performance benchmarks, selects the best performers, and iterates — much like natural selection. Over many generations, this process converges on merged models that inherit the best qualities of their parents. Unlike simple model averaging or manual merging techniques that have existed in the open-source community for years, Sakana AI's method automates and optimizes the entire search process.

Critically, this approach can combine models with different specializations. A model fine-tuned for Japanese language understanding can be merged with one optimized for mathematical reasoning, producing a bilingual model with strong math capabilities — something that would normally require curating a massive multilingual math dataset and training from the ground up.

Sakana AI's Rise From Stealth to $2 Billion Valuation

Sakana AI's trajectory has been nothing short of meteoric. Founded in mid-2023 by Llion Jones and David Ha, the company emerged from stealth with a bold thesis: that the future of AI lies not in building ever-larger monolithic models, but in intelligently combining smaller, specialized ones.

The startup's pedigree immediately turned heads. Jones was one of 8 co-authors of the 2017 'Attention Is All You Need' paper at Google Brain, which introduced the transformer architecture — the foundation underlying virtually every modern large language model from GPT-4 to Claude to Gemini. Ha brought deep expertise in evolutionary computation and generative models from his time at Google Brain and Stability AI.

Investor interest followed quickly:

• Seed round: Approximately $30 million from Lux Capital, Khosla Ventures, and others in late 2023
• Series A: Over $300 million led by SoftBank, with participation from NVIDIA, Samsung, and others
• Valuation: Reported at approximately $2 billion, making Sakana AI one of the most valuable AI startups in Asia
• Team growth: The company has expanded to over 50 researchers, drawing talent from Google DeepMind, Meta FAIR, and leading Japanese universities

SoftBank CEO Masayoshi Son has been vocal about his belief that AI represents the defining investment opportunity of the next decade. Sakana AI fits neatly into his broader strategy of positioning Japan as a major player in the global AI race, particularly as the Japanese government has committed over $13 billion to AI and semiconductor infrastructure.

Benchmark Results Show Promising Performance Gains

The real test of any AI research breakthrough lies in measurable results. Sakana AI's evolutionary merging technique has produced several noteworthy outcomes that distinguish it from the broader model merging community.

In published research, the team demonstrated that their evolved models consistently outperform their constituent source models across multiple benchmarks. For instance, a merged model combining a Japanese language model with an English math-reasoning model achieved state-of-the-art performance on Japanese math benchmarks — surpassing both parent models and even some models specifically trained for this task.

Compared to traditional fine-tuning approaches, the evolutionary method offers several quantifiable advantages:

• Compute savings: Model merging requires only evaluation compute (running inference on test sets), not training compute. This can reduce costs by 90% or more compared to full fine-tuning
• Speed: Merging experiments that produce competitive models can complete in hours rather than weeks
• No data requirements: Unlike fine-tuning, the merging process doesn't require curated training datasets for the target domain
• Scalability: The technique can be applied to any combination of open-weight models, creating a combinatorial explosion of possibilities

However, experts caution that evolutionary merging is not a replacement for foundational pre-training. The technique works best when high-quality source models already exist. It's an optimization and combination strategy, not a way to create capabilities from nothing.

Industry Context: A Shift Toward Efficient AI Development

Sakana AI's breakthrough arrives at a pivotal moment in the AI industry. The cost of training frontier models has skyrocketed — OpenAI's GPT-4 reportedly cost over $100 million to train, and next-generation models are expected to cost $1 billion or more. This escalating expense has created a growing divide between a handful of well-funded labs and everyone else.

The open-source AI community has already embraced rudimentary model merging as a workaround. Platforms like Hugging Face host thousands of community-merged models, with tools like mergekit enabling hobbyists to combine models using techniques such as SLERP (Spherical Linear Interpolation), TIES, and DARE. Some of the top-performing models on the Open LLM Leaderboard have been community merges rather than officially trained models.

Sakana AI's contribution elevates this practice from artisanal experimentation to rigorous, automated optimization. By applying evolutionary search to the merging process, they remove the guesswork and manual parameter tuning that currently limits the technique's reliability.

This fits into a broader industry trend toward efficient AI development. Companies like Mistral AI in France have demonstrated that smaller, well-architected models can compete with much larger ones. China's DeepSeek made waves with cost-efficient training methods. Apple and Microsoft have invested heavily in on-device AI that prioritizes efficiency over raw scale.

What This Means for Developers and Businesses

The practical implications of evolutionary model merging extend far beyond academic interest. For developers and enterprises, this technology could unlock several concrete benefits.

Custom model creation becomes accessible. Companies that need domain-specific AI — say, a legal AI that also handles multilingual queries — could potentially merge existing specialized models rather than commissioning expensive custom training runs. This dramatically lowers the barrier to entry for AI customization.

Rapid prototyping accelerates. Product teams could experiment with merged model variants in hours rather than waiting weeks for fine-tuning jobs to complete. This enables faster iteration cycles and more experimentation with different capability combinations.

Cost structures improve. For mid-size companies that can't afford $10 million training runs, evolutionary merging offers a path to competitive model performance at a fraction of the cost. This could level the playing field between AI startups and well-funded incumbents.

However, challenges remain. Merged models can inherit biases and failure modes from their parent models in unpredictable ways. Quality assurance and safety evaluation for evolved models will require new frameworks and testing methodologies that don't yet exist in standardized form.

Looking Ahead: The Future of Nature-Inspired AI

Sakana AI's evolutionary approach represents just the beginning of what nature-inspired methods could contribute to AI development. The company's name itself — 'sakana' means 'fish' in Japanese — reflects its philosophy of drawing inspiration from the collective intelligence found in natural systems like schools of fish or swarms of insects.

Several developments are worth watching in the coming months:

Scaling the approach. As open-source models proliferate — Meta alone has released dozens of Llama variants — the combinatorial space for evolutionary merging grows exponentially. Sakana AI is reportedly developing more sophisticated search algorithms to navigate this expanding landscape efficiently.

Enterprise adoption. SoftBank's deep corporate relationships across Japan and Asia could accelerate enterprise deployment of Sakana AI's technology, particularly in sectors like automotive, robotics, and manufacturing where Japan holds competitive advantages.

Competitive response. Major AI labs are likely watching Sakana AI's work closely. If evolutionary merging proves consistently effective, expect Google DeepMind, Meta FAIR, and others to incorporate similar techniques into their model development pipelines.

Regulatory implications. As AI governance frameworks evolve in the EU, US, and Japan, the question of how to evaluate and certify merged models — which may combine components from different developers with different safety testing — will become increasingly important.

Sakana AI has positioned itself at the intersection of two powerful trends: the growing open-source model ecosystem and the urgent need for more compute-efficient AI development methods. If evolutionary model merging fulfills its promise, the startup could help usher in an era where creating powerful, specialized AI systems no longer requires the resources of a tech giant — just clever recombination of what already exists.