Sakana AI Pioneers Evolutionary Model Merging

Sakana AI Introduces a New Paradigm for Building LLMs

Japan-based startup Sakana AI has developed a groundbreaking approach called Evolutionary Model Merging that combines existing large language models into new, more capable systems — without the enormous computational costs of training from scratch. The technique, which leverages evolutionary algorithms to discover optimal ways to merge model parameters and architectures, represents a significant shift in how the AI industry thinks about building foundation models.

While companies like OpenAI, Google, and Anthropic pour billions of dollars into training ever-larger models from the ground up, Sakana AI's approach suggests there may be a far more efficient path. The Tokyo-headquartered company, founded by former Google Brain researchers, is positioning itself as a leader in what it calls 'nature-inspired' AI development.

Key Takeaways

• Sakana AI uses evolutionary algorithms to merge multiple pre-trained LLMs into superior new models
• The approach dramatically reduces computational costs compared to training models from scratch
• Merged models have demonstrated strong performance across multiple benchmarks, including math reasoning and Japanese language tasks
• The company was founded by Llion Jones, a co-author of the landmark 'Attention Is All You Need' transformer paper, and David Ha, formerly of Google Brain
• Sakana AI raised approximately $300 million in funding, reaching a valuation near $2 billion
• The technique opens the door to rapid, automated model development without massive GPU clusters

How Evolutionary Model Merging Actually Works

Model merging itself is not entirely new. The open-source AI community has experimented with combining weights from different fine-tuned models for some time, often producing surprisingly capable results. What Sakana AI brings to the table is automation and optimization of this process through evolutionary search.

Traditional model merging relies on human intuition and manual experimentation to determine which models to combine and how to blend their parameters. Sakana AI's method replaces this guesswork with an evolutionary algorithm — a computational technique inspired by natural selection. The system generates a population of candidate merged models, evaluates their performance, and iteratively selects and recombines the best performers over many generations.

The approach operates on 2 key levels. First, it optimizes the parameter space, determining the ideal mixing ratios for weights from different source models across each layer. Second, it explores the architecture space, experimenting with how layers from different models can be stacked and rearranged to form entirely new network topologies. This dual optimization is what sets Sakana AI's work apart from simpler merging techniques like SLERP (Spherical Linear Interpolation) or TIES (TrIm, Elect Sign & Merge) that the open-source community commonly uses.

Impressive Results Without Billion-Dollar Budgets

The results Sakana AI has demonstrated are striking, particularly given the minimal resources required. The company's evolutionary merging process produced several notable models that outperform their individual source components.

One standout result is a Japanese math reasoning model that was created by merging a large Japanese language model with an English-focused math reasoning model. Neither source model excelled at both tasks simultaneously, but the merged model achieved strong performance in Japanese mathematical reasoning — a capability that neither parent model possessed on its own.

Key performance highlights include:

• A merged Japanese LLM that outperformed several individually trained Japanese language models on standard benchmarks
• A vision-language model created entirely through merging, with no additional training data required
• Models that demonstrate cross-lingual transfer, combining language capabilities from one model with task-specific skills from another
• Total compute costs that are a fraction of what conventional training would demand — potentially under $100 in cloud computing credits for some configurations
• Competitive benchmark scores against models that cost millions of dollars to train

Compared to training a model like Meta's Llama 3 or Google's Gemma, which require thousands of GPUs running for weeks or months, Sakana AI's evolutionary search can be conducted on a single node with modest hardware in a matter of hours or days.

The Founders Bring Deep Transformer Expertise

Sakana AI was co-founded in 2023 by 2 prominent figures in the AI research world. Llion Jones is one of the 8 co-authors of the 2017 paper 'Attention Is All You Need,' which introduced the transformer architecture — the foundation of virtually every modern LLM including GPT-4, Claude, and Gemini. David Ha previously served as the head of research at Stability AI and spent years at Google Brain working on evolutionary computation and creative AI systems.

The combination of deep transformer expertise and evolutionary computation background is no coincidence. Sakana AI's core thesis is that the AI industry's reliance on brute-force scaling — simply making models bigger and training them on more data — is not the only viable strategy. The company's name, which means 'fish' in Japanese, reflects its philosophy of looking to collective natural intelligence, like schools of fish, for inspiration.

The startup's $300 million funding round, which included investment from prominent backers like Lux Capital, Khosla Ventures, and NTT Group, signals strong investor confidence in this alternative approach. At a roughly $2 billion valuation, Sakana AI is one of the most valuable AI startups in Japan and increasingly a player on the global stage.

Why This Matters for the Broader AI Industry

Sakana AI's work arrives at a critical inflection point for the AI industry. The cost of training frontier models has skyrocketed — estimates suggest GPT-4 cost over $100 million to train, and next-generation models could exceed $1 billion. This escalating expense threatens to concentrate AI development among a handful of deep-pocketed tech giants.

Evolutionary model merging offers a potential democratization pathway. If powerful new models can be created by intelligently combining existing open-source models, then smaller companies, academic institutions, and developers in emerging markets gain meaningful access to cutting-edge AI capabilities. This aligns with a broader trend in the open-source AI community, where platforms like Hugging Face already host thousands of fine-tuned model variants that could serve as raw material for evolutionary merging.

The implications extend beyond cost savings:

• Rapid specialization: Organizations could quickly create domain-specific models by merging a general-purpose LLM with a specialized one
• Reduced environmental impact: Less compute means lower energy consumption and carbon emissions
• Accelerated innovation cycles: New models could be produced in hours instead of months
• Accessibility: Smaller teams and startups can compete with well-funded labs
• Composability: The AI ecosystem could evolve toward modular, mix-and-match model development

This approach also raises interesting questions about intellectual property and model licensing. If a merged model inherits capabilities from multiple source models, each with different licenses, the legal landscape becomes complex.

Challenges and Limitations Remain

Despite its promise, evolutionary model merging is not a silver bullet. Several significant challenges need to be addressed before it can replace conventional training at the frontier.

Benchmark performance on merged models, while impressive relative to their cost, still generally falls short of purpose-built frontier models like GPT-4o or Claude 3.5 Sonnet. The technique excels at combining existing capabilities in novel ways but cannot yet generate fundamentally new capabilities that none of the source models possess.

Scalability is another open question. Most demonstrated merges have involved models in the 7B to 70B parameter range. Whether evolutionary merging can produce results competitive with 400B+ parameter frontier models remains to be seen. There are also concerns about evaluation reliability — merged models may perform well on specific benchmarks while exhibiting unexpected weaknesses in real-world deployment scenarios.

The theoretical understanding of why model merging works at all is still incomplete. Researchers hypothesize that different fine-tuned models occupy nearby regions in parameter space, making linear interpolation between them surprisingly effective. But a rigorous mathematical framework for predicting merge outcomes remains elusive.

Looking Ahead: The Future of Efficient AI Development

Sakana AI's evolutionary model merging points toward a future where AI development looks less like a single massive training run and more like an ecosystem of evolving, recombining models. This vision aligns with emerging research trends around mixture-of-experts architectures, modular neural networks, and collaborative AI development.

In the near term, expect to see Sakana AI release more merged models and potentially open-source their evolutionary search tools. The company has indicated interest in applying its techniques to multimodal models that combine text, image, and video capabilities — a natural extension of cross-domain merging.

The broader industry is taking notice. Several research groups have published follow-up work exploring variations on evolutionary merging, and major open-source communities are integrating merging tools into their workflows. Platforms like MergeKit, which already facilitate manual model merging, could eventually incorporate evolutionary optimization techniques inspired by Sakana AI's research.

For developers and businesses, the practical message is clear: the barrier to creating specialized, high-performing AI models is dropping rapidly. Within the next 12 to 18 months, evolutionary model merging could become a standard tool in the AI practitioner's toolkit — enabling organizations of all sizes to build custom models that rival systems costing orders of magnitude more to develop.

Sakana AI's work is a compelling reminder that in AI, as in nature, the most successful strategies are not always about raw power. Sometimes, the smartest approach is learning to combine what already exists in new and unexpected ways.