Build Real-Time AI Search With Elasticsearch

Why Traditional Search Falls Short in the AI Era

Real-time AI search engines are rapidly replacing keyword-based systems across the tech industry, and the combination of Elasticsearch with modern embedding models has emerged as one of the most practical paths to production-ready semantic search. Companies like Airbnb, Spotify, and Uber already leverage vector-based search to deliver results that understand user intent rather than simply matching keywords.

This guide walks through the architecture, tooling, and implementation strategies for building a real-time AI-powered search engine using Elasticsearch 8.x and popular embedding models like OpenAI's text-embedding-3-small, Cohere Embed v3, and open-source alternatives from Hugging Face.

Key Takeaways

• Elasticsearch 8.x natively supports dense vector fields and approximate nearest neighbor (ANN) search via HNSW algorithms
• Combining traditional BM25 scoring with vector similarity — known as hybrid search — delivers 15-30% better relevance than either approach alone
• OpenAI's text-embedding-3-small costs just $0.02 per 1 million tokens, making it viable for large-scale indexing
• Open-source models like all-MiniLM-L6-v2 offer zero-cost embedding generation with sub-10ms latency
• Real-time indexing pipelines can process 5,000-10,000 documents per second with proper batching
• The entire stack can run on a single node for prototyping or scale horizontally for production workloads

Understanding the Core Architecture

Semantic search works fundamentally differently from traditional full-text search. Instead of matching keywords in an inverted index, it converts both documents and queries into high-dimensional vectors — numerical representations that capture meaning. Documents with similar meanings cluster together in vector space, regardless of the specific words used.

The architecture for a real-time AI search engine consists of 4 primary components. First, an ingestion pipeline that preprocesses and chunks incoming documents. Second, an embedding service that converts text into vectors. Third, an Elasticsearch cluster that stores and indexes both the raw text and vector representations. Fourth, a query orchestrator that handles hybrid retrieval and re-ranking.

Unlike purpose-built vector databases such as Pinecone or Weaviate, Elasticsearch offers a significant advantage: it combines vector search with its battle-tested full-text capabilities in a single system. This eliminates the operational complexity of managing separate infrastructure for keyword and semantic search.

Setting Up Elasticsearch for Vector Search

Elasticsearch 8.x introduced native support for dense vector fields and kNN search, making it a first-class citizen for AI-powered retrieval. The setup process begins with configuring an index mapping that accommodates both traditional text fields and vector embeddings.

The index mapping requires careful planning. You need to define the dense_vector field type with the correct dimensionality matching your chosen embedding model. OpenAI's text-embedding-3-small produces 1,536-dimensional vectors, while all-MiniLM-L6-v2 generates 384-dimensional vectors. Smaller dimensions mean faster search but potentially less nuanced representations.

Key configuration parameters for the vector field include:

• dims: Must match your embedding model's output dimensionality exactly
• index: Set to true to enable ANN search rather than brute-force comparison
• similarity: Choose between cosine, dot_product, or l2_norm based on your model's training objective
• index_options.type: Use 'hnsw' for the best balance of speed and recall
• index_options.m: Controls graph connectivity — higher values (16-32) improve recall at the cost of memory
• index_options.ef_construction: Set between 100-200 for production-quality index builds

For most use cases, cosine similarity paired with HNSW indexing delivers the best out-of-the-box performance. The default HNSW parameters work well for datasets under 10 million documents.

Choosing the Right Embedding Model

Selecting an embedding model is arguably the most consequential decision in the entire pipeline. The model determines the quality ceiling for your search results, and switching models later requires re-indexing your entire corpus.

OpenAI's text-embedding-3-small has become the default choice for many teams. At $0.02 per million tokens, it offers strong multilingual performance with 1,536 dimensions. Its larger sibling, text-embedding-3-large, produces 3,072-dimensional vectors and scores higher on benchmarks like MTEB, but at $0.13 per million tokens — a 6.5x cost increase that many teams find hard to justify.

Cohere Embed v3 is a strong alternative, particularly for retrieval-specific tasks. Unlike general-purpose embeddings, Cohere allows you to specify an input_type parameter — 'search_document' for indexing and 'search_query' for queries — which can improve relevance by 5-10% compared to symmetric embedding models.

For teams prioritizing data privacy or cost elimination, open-source models from Hugging Face provide compelling options:

• all-MiniLM-L6-v2: 384 dimensions, 80ms inference on CPU, ideal for prototyping
• BGE-large-en-v1.5: 1,024 dimensions, competitive with commercial APIs on English-language benchmarks
• E5-mistral-7b-instruct: 4,096 dimensions, state-of-the-art quality but requires GPU inference
• GTE-large: 1,024 dimensions, strong performance on short-text retrieval tasks

Running open-source models locally with frameworks like Sentence Transformers or FastEmbed eliminates per-token costs entirely. A single NVIDIA T4 GPU ($0.50/hour on cloud providers) can process roughly 500 embeddings per second with BGE-large.

Building the Real-Time Ingestion Pipeline

Real-time indexing requires a streaming architecture that processes documents as they arrive rather than in batch jobs. Apache Kafka or Amazon Kinesis typically serves as the message backbone, feeding documents into an embedding and indexing service.

The ingestion pipeline follows a 5-step flow. Documents first enter a preprocessing stage where HTML is stripped, text is normalized, and metadata is extracted. Next, a chunking strategy splits long documents into segments of 256-512 tokens with 50-token overlap to preserve context across boundaries. The chunks then pass through the embedding service, which batches requests for efficiency — OpenAI's API accepts up to 2,048 texts per batch call.

After embedding generation, the system constructs Elasticsearch bulk index requests containing both the original text, metadata, and the vector representation. Bulk indexing with batches of 500-1,000 documents dramatically outperforms single-document indexing, often achieving 10x higher throughput.

Latency optimization matters for real-time systems. The total pipeline latency from document arrival to searchability should target under 2 seconds. The embedding step typically dominates at 100-500ms for API-based models. Setting Elasticsearch's refresh_interval to 1 second (the default) ensures new documents become searchable almost immediately.

Implementing Hybrid Search for Maximum Relevance

Hybrid search — combining BM25 keyword matching with vector similarity — consistently outperforms either approach in isolation. Elasticsearch 8.x supports this through the Reciprocal Rank Fusion (RRF) algorithm, which merges results from multiple retrieval strategies without requiring manual weight tuning.

The query flow works as follows. When a user submits a search query, the orchestrator simultaneously fires 2 sub-queries against Elasticsearch: a traditional match query using BM25 scoring and a kNN query using the embedded query vector. RRF then combines both result sets by assigning each document a score based on its rank position in each list, using the formula: score = 1 / (k + rank), where k is typically set to 60.

This approach handles edge cases that pure vector search misses. Exact product names, error codes, and technical identifiers benefit enormously from keyword matching. Meanwhile, conceptual queries like 'how to fix slow database performance' benefit from semantic understanding. Hybrid search captures both.

Benchmarks from Elasticsearch Labs show that RRF hybrid search improves nDCG@10 by 15-30% over pure vector search across standard information retrieval datasets. The computational overhead is minimal — adding the BM25 sub-query typically adds less than 5ms to total query latency.

Optimizing Performance at Scale

Production deployments require careful attention to resource allocation, caching, and query optimization. Vector search is inherently more memory-intensive than traditional search because HNSW graphs must reside in memory for fast retrieval.

Memory planning follows a straightforward formula. Each vector consumes 4 bytes per dimension. A corpus of 10 million documents with 1,536-dimensional vectors requires approximately 57 GB of RAM just for the vectors, plus overhead for the HNSW graph structure. Choosing a smaller model like all-MiniLM-L6-v2 (384 dimensions) reduces this to roughly 14 GB.

Key optimization strategies include:

• Quantization: Elasticsearch supports byte and int8 quantization, reducing memory usage by 4x with minimal recall loss (typically under 2%)
• Filtered kNN: Apply metadata filters before vector search to reduce the candidate set and improve both speed and relevance
• Caching embedding results: Store query embeddings in Redis with a 5-minute TTL to avoid redundant API calls for repeated searches
• Index sharding: Distribute vectors across 2-4 shards for datasets exceeding 5 million documents
• Result caching: Cache the top-100 results for popular queries, reducing P99 latency from 200ms to under 10ms

For most applications serving under 100 queries per second, a 3-node Elasticsearch cluster with 32 GB RAM per node handles 10-20 million documents comfortably.

What This Means for Development Teams

The convergence of Elasticsearch's vector capabilities with affordable embedding models has democratized AI search in a meaningful way. Teams no longer need to choose between the operational maturity of Elasticsearch and the semantic power of vector search — they get both in a single platform.

Cost barriers have largely disappeared. A startup can build a production-quality semantic search engine for under $500/month in infrastructure costs, including embedding generation. Compared to 2022, when similar capabilities required specialized vector databases, custom infrastructure, and expensive embedding APIs, the total cost of ownership has dropped by roughly 70%.

Developers familiar with the Elastic ecosystem can leverage existing monitoring, security, and management tooling. The learning curve primarily involves understanding embedding model selection and vector field configuration — skills that transfer directly to other vector search platforms.

Looking Ahead: The Future of AI Search

Late-interaction models like ColBERT and ColPali represent the next frontier, offering higher relevance than single-vector representations by preserving token-level information. Elasticsearch has signaled interest in supporting these architectures natively in future releases.

The integration of retrieval-augmented generation (RAG) with real-time search engines is accelerating. By combining Elasticsearch's hybrid search with large language models like GPT-4o or Claude 3.5, developers can build systems that not only find relevant documents but synthesize coherent answers from them.

Expect embedding model costs to continue falling. OpenAI has already reduced embedding prices by 5x since 2023, and open-source alternatives are closing the quality gap rapidly. By mid-2025, the distinction between commercial and open-source embedding quality will likely be negligible for most English-language use cases.

The bottom line: building a real-time AI search engine is no longer a moonshot engineering project. With Elasticsearch 8.x, a well-chosen embedding model, and the hybrid search patterns outlined above, a small team can ship production-grade semantic search in weeks rather than months.