Tencent Open-Sources OpenSearch-VL for Multimodal AI Search

Tencent Tackles Multimodal Search Agent Training With Open-Source Framework

Tencent Hunyuan has partnered with UCLA and the Chinese University of Hong Kong to release OpenSearch-VL, a fully open-source training framework designed to overcome the biggest bottleneck in building multimodal search AI agents: the lack of high-quality training data. The research paper, published on arXiv on May 6, introduces a complete pipeline — from data construction and tool integration to training algorithms — that enables developers to build frontier-level deep search agents capable of processing images, text, and other modalities simultaneously.

Unlike proprietary systems from commercial AI labs that keep their data sources, filtering criteria, and tool-use trajectories locked behind closed doors, OpenSearch-VL provides full transparency. This marks a significant step toward democratizing advanced multimodal AI research for the broader community.

Key Takeaways

• Open-source multimodal training: OpenSearch-VL offers a complete, reproducible pipeline for training deep search agents
• Reinforcement learning approach: The framework uses RL techniques to train agents that actively call external tools like search engines and image processors
• High-quality data pipeline: Introduces Wikipedia path sampling and fuzzy entity rewriting to produce robust datasets including SearchVL-SFT-36k
• Collaboration across institutions: Joint effort between Tencent Hunyuan, UCLA, and the Chinese University of Hong Kong
• Addresses industry bottleneck: Directly targets the training data scarcity problem that has slowed multimodal search agent development
• Frontier performance: Aims to match or exceed capabilities of proprietary commercial systems

What Are Multimodal Search Agents and Why Do They Matter?

Multimodal search agents represent a new class of AI systems that go far beyond traditional chatbots or image classifiers. These agents can accept inputs across multiple modalities — images, text, documents, and more — and then autonomously invoke external tools such as search engines, image analysis utilities, and knowledge bases to perform multi-step reasoning.

The goal is solving knowledge-intensive visual question answering tasks. Imagine uploading a photo of an obscure historical building and asking an AI to identify it, trace its architectural lineage, and explain its cultural significance. A multimodal search agent would analyze the image, search the web for matching structures, cross-reference historical databases, and synthesize a comprehensive answer — all without human intervention.

This capability has enormous implications for industries ranging from e-commerce product search to medical imaging diagnostics and cultural heritage preservation. However, building these agents has remained largely the domain of well-resourced commercial labs, primarily because the training data and methodologies required have been proprietary.

The Training Data Bottleneck That Has Stalled Progress

The OpenSearch-VL research team identifies a critical problem: the single biggest barrier to advancing multimodal search agents is not compute power or model architecture — it is high-quality training data. Current state-of-the-art systems are overwhelmingly developed by commercial companies that treat their data pipelines as trade secrets.

This creates several cascading problems for the research community:

• Irreproducibility: Without access to training data and filtering standards, independent researchers cannot replicate results from leading commercial systems
• Limited systematic study: The lack of standardized open datasets prevents rigorous comparative analysis across approaches
• Slow iteration: Academic labs and smaller companies cannot build on proprietary foundations, forcing them to start from scratch
• Homogeneous development: Innovation concentrates in a handful of large corporations rather than benefiting from diverse global contributions

The team argues that this data bottleneck has created an asymmetry in the field. While large language model research has benefited enormously from open datasets and open-weight models — think Meta's Llama series or Mistral's releases — multimodal search agent development has lagged behind in openness. OpenSearch-VL aims to correct this imbalance.

How OpenSearch-VL Builds Better Training Data

At the heart of OpenSearch-VL is an innovative data construction pipeline that produces high-quality training samples while deliberately avoiding common pitfalls. The framework introduces 2 key techniques that set it apart from prior approaches.

First, Wikipedia path sampling generates complex, multi-hop queries by tracing connections between Wikipedia articles. Instead of creating simple single-step questions, the pipeline follows relational paths through Wikipedia's knowledge graph to construct questions that require genuine multi-step reasoning and evidence gathering.

Second, fuzzy entity rewriting addresses a subtle but critical problem known as 'retrieval shortcuts.' When training data contains exact entity names that can be trivially matched to search results, agents learn to take shortcuts rather than developing genuine reasoning capabilities. By deliberately introducing ambiguity into entity references, OpenSearch-VL forces agents to develop more robust search and verification strategies.

These techniques together produce the SearchVL-SFT-36k dataset — a collection of 36,000 high-quality supervised fine-tuning samples specifically designed for multimodal search agent training. The dataset represents one of the first large-scale, publicly available resources tailored to this task.

Reinforcement Learning Powers Autonomous Tool Use

Beyond data construction, OpenSearch-VL employs reinforcement learning (RL) to train agents that can autonomously decide when and how to use external tools. This is a crucial distinction from simpler approaches that rely solely on supervised fine-tuning.

In the RL framework, the agent learns through trial and error to:

• Determine when an image or text query requires external search
• Formulate effective search queries based on visual and textual inputs
• Evaluate and filter search results for relevance and accuracy
• Decide when sufficient evidence has been gathered to formulate an answer
• Synthesize information from multiple sources and modalities into coherent responses

This approach mirrors the methodology that has driven recent breakthroughs in reasoning models. Companies like OpenAI with o1 and DeepSeek with DeepSeek-R1 have demonstrated that RL can dramatically improve an AI system's ability to perform complex, multi-step reasoning. OpenSearch-VL applies similar principles but extends them into the multimodal search domain, where agents must reason across both visual and textual information while actively interacting with external tools.

How OpenSearch-VL Compares to Existing Solutions

The multimodal AI search space has grown increasingly competitive in 2025. Perplexity AI has popularized AI-powered search for consumers, while Google's Gemini models integrate multimodal understanding with search capabilities. However, these systems remain proprietary and closed-source.

On the open-source side, projects like LLaVA and InternVL have advanced multimodal understanding, but they primarily focus on perception and reasoning rather than active tool use and search. OpenSearch-VL occupies a unique position by combining multimodal perception with autonomous search and tool-use capabilities in a fully open framework.

The key differentiators include:

• End-to-end openness: Unlike commercial alternatives, every component from data to training code is publicly available
• Tool-use integration: Goes beyond passive multimodal understanding to active search and verification
• Scalable data pipeline: The Wikipedia-based data construction method can generate training data at scale without manual annotation
• RL-based training: Leverages reinforcement learning for more robust and generalizable agent behavior compared to supervised-only approaches

Industry Context: The Open-Source AI Arms Race Intensifies

Tencent's release of OpenSearch-VL fits into a broader pattern of Chinese tech giants aggressively contributing to the open-source AI ecosystem. In 2024 and 2025, companies including Alibaba (Qwen series), DeepSeek, and ByteDance have released increasingly powerful open models that rival or exceed proprietary Western alternatives in specific benchmarks.

This trend has significant implications for the global AI landscape. Open-source releases like OpenSearch-VL lower barriers to entry, enabling startups, academic researchers, and developers in smaller organizations to build sophisticated AI applications without massive R&D budgets. They also create competitive pressure on proprietary systems to demonstrate clear value above and beyond what is freely available.

For Western developers and companies, the proliferation of high-quality open-source AI tools from Chinese institutions presents both an opportunity and a strategic consideration. The technology is freely available and often state-of-the-art, but questions around long-term support, documentation quality, and integration with Western tech stacks remain relevant factors in adoption decisions.

What This Means for Developers and Businesses

OpenSearch-VL's release has immediate practical implications for several groups. AI researchers now have a reproducible baseline for multimodal search agent development, enabling apples-to-apples comparisons and systematic improvements. Application developers can leverage the framework to build specialized search tools for domains like e-commerce, healthcare, and education without starting from zero.

Enterprise teams exploring AI-powered knowledge management and visual search systems can evaluate OpenSearch-VL as a foundation for custom solutions. The fully open nature of the project means organizations can audit, modify, and deploy the technology according to their specific requirements and compliance needs.

The SearchVL-SFT-36k dataset alone represents significant value. High-quality, task-specific training data remains one of the most expensive and time-consuming components of any AI development project. Having a publicly available, well-constructed dataset for multimodal search training could accelerate development timelines by weeks or months.

Looking Ahead: The Future of Multimodal AI Search

OpenSearch-VL represents an early but important step in what is likely to become a rapidly evolving field. As multimodal AI models continue to improve in their base capabilities — better image understanding, longer context windows, more nuanced reasoning — the potential for sophisticated search agents will grow correspondingly.

Several trends are worth watching in the coming months. First, expect other research groups and companies to build on OpenSearch-VL's open foundation, potentially producing improved datasets, training techniques, and agent architectures. Second, the integration of multimodal search capabilities into consumer-facing products will likely accelerate as frameworks like this lower the technical barriers.

Finally, the competition between open-source and proprietary approaches in this space will intensify. If open frameworks can match or approach the performance of commercial systems, it could reshape the economics of AI-powered search — shifting competitive advantage from data hoarding to application-layer innovation and user experience design.

The research paper and associated code are available on arXiv and the project's GitHub repository. Developers interested in multimodal AI search should closely monitor this project as the community begins to build on its foundation.