RAG vs Traditional Search: What Changed, What Did Not, and Why BM25 Is Not Dead

In 2016, when a user searched your company's internal wiki for "password reset steps," they got a list of ten documents sorted by keyword relevance. They clicked the first result, skimmed three paragraphs, found the steps, and did the thing.

In 2026, that same query goes into a RAG system. The system retrieves the relevant sections from your documentation. The LLM reads them. It returns: "To reset your password, go to Account Settings, click Security, then select Reset Password. You will receive an email within two minutes."

Same underlying information. Different interface. Different outcome.

The retrieval mechanics have more in common than most people think. What changed is what happens after retrieval - and that change has enormous implications for when you use which system.

What Traditional Search Actually Does

Traditional search is a retrieval and ranking problem. You have a corpus of documents. A user submits a query. The search engine scores every document for relevance to that query and returns a ranked list.

BM25 - Best Match 25 - is the algorithm that has dominated that ranking problem for two decades. It is the default ranking function in Elasticsearch, Apache Solr, and OpenSearch. It evaluates relevance using three signals: how often the query terms appear in the document (term frequency), how rare those terms are across the entire corpus (inverse document frequency), and document length as a normalization factor.

BM25 sees text as a bag of tokens. It has no understanding of language, meaning, or intent. It matches words, not concepts. A document that says "cardiac arrest treatment protocol" will rank low for the query "heart failure emergency procedures" even though those phrases describe the same clinical situation. The words do not overlap.

Elasticsearch has powered over 80% of the world's search infrastructure on the strength of this model. Fast, interpretable, scalable to billions of documents, cheap to run. For use cases where users know the right words and exact matching is what they need - log search, compliance document lookup, product SKU search - BM25 is still hard to beat.

What BM25 cannot do is understand that "money back guarantee" and "refund policy" describe the same thing, or that a question about "what to do if my payment fails" should retrieve a document titled "billing error resolution." For that, you need semantic understanding.

What Semantic Search Adds

Semantic search converts queries and documents into dense vector embeddings - numerical representations that capture meaning rather than surface form. Documents about similar concepts cluster together in vector space regardless of the specific words used. A query about "heart failure emergency procedures" retrieves documents about "cardiac arrest treatment protocol" because those phrases land near each other in the embedding space.

Vector search can match "cardiac arrest" to a document about "heart failure" even though none of the words overlap, because the embedding model has learned that these concepts live close together in semantic space.

This is powerful. It is also not a replacement for keyword search. Semantic search has a mirror-image weakness to BM25. Where BM25 misses synonyms and paraphrases, semantic search misses exact terms. A product code like AX-7200-PRO, an error string like ECONNREFUSED, or a statute number like § 1983 may embed poorly relative to their meaning in context. The embedding model tokenizes these identifiers into subword pieces that do not cluster near the exact string a user searches for.

This is the core tension that drove the development of hybrid retrieval - and it is the reason BM25 is more alive than ever in 2026.

Why Traditional Retrieval Still Matters

Neither retrieval style covers every query. This comparison owns that decision-level distinction; Hybrid Search for RAG owns how a system combines them.

What RAG Adds on Top of Retrieval

RAG does not replace the retrieval layer. It extends it.

Traditional search returns a list of documents ranked by relevance. The user reads those results. RAG takes the top-ranked results, passes them to a language model as context, and asks the model to synthesize a direct answer from what it retrieved. The user receives an answer, not a list of documents.

Traditional Search Flow
+---------------------------+
|  User Query               |
|          |                |
|          v                |
|  BM25 / Vector Index      |
|          |                |
|          v                |
|  Ranked Document List     |  <-- user reads this and extracts answer
+---------------------------+

RAG Flow
+---------------------------+
|  User Query               |
|          |                |
|          v                |
|  Search / Retrieval Index |
|          |                |
|          v                |
|  Reranker                 |
|          |                |
|          v                |
|  Top-k Chunks (context)   |
|          |                |
|          v                |
|  LLM Generation           |
|          |                |
|          v                |
|  Synthesized Answer       |  <-- user reads this
|  + Source Citations       |
+---------------------------+

The retrieval step inside RAG is identical in structure to traditional search. What changes is the terminal step. Instead of presenting documents for the user to read, RAG feeds those documents to a model that reads them on the user's behalf and produces a direct, synthesized, cited answer.

This has consequences in both directions.

RAG wins when: The user needs an answer, not a document. When the relevant information is spread across multiple sections of multiple documents and the user would need to read and cross-reference all of them to construct the answer themselves. When the query is conversational and contextual, building on previous turns. When traceability and citation of sources matter for compliance.

Traditional search wins when: The user needs the source document, not a summary. When the corpus has very high query volume and sub-second response is required. When exact document retrieval is the goal - a compliance audit that requires the specific PDF, not a model's interpretation of it. When the cost per query of running an LLM at scale is unjustifiable for the value it adds.

Why BM25 Still Belongs in Modern Retrieval

Semantic retrieval did not make traditional search obsolete. BM25 remains valuable because exact identifiers, quoted phrases, names, error strings, and regulated terminology often need literal matching rather than conceptual similarity.

That does not mean every RAG system must use the same combined pipeline. It means lexical retrieval is still a valid component when the query mix demands it. The mechanics of running BM25 and dense search together, merging their results, configuring RRF, and evaluating fusion belong in Hybrid Search for RAG.

Where Traditional Search Still Wins Outright

BM25 keyword matching is fast and precise: 10 to 50 milliseconds even on 10GB document collections. A basic RAG pipeline adds the embedding computation plus the LLM generation step, pushing total end-to-end latency to 1 to 3 seconds for a straightforward query. With reranking, 1.5 to 3.5 seconds. With agentic multi-hop retrieval, 5 to 15 seconds.

For use cases where that latency gap matters, and where the generation step adds no value, traditional search is the right architecture.

The Migration Path: Elasticsearch to RAG

Most organizations already have an Elasticsearch or Solr deployment that handles their internal search. The question is not "should we replace it with RAG?" It is "how do we layer RAG generation on top of what we already have?"

Teams running existing search infrastructure can add RAG without replacing the retrieval layer. The search system continues returning relevant source passages; an LLM consumes those passages and generates an answer. This preserves exact-match, filtering, and operational-search workloads that already work.

Existing Elasticsearch Index
          |
          v
Ranked Results (with source metadata)
          |
          v
LLM Generation Layer
          |
          v
Synthesized Answer + Citations

Organizations that already run Elasticsearch for structured operational search can keep it for those workloads and add a RAG layer over the unstructured documentation subset - the product manuals, policy documents, and knowledge base articles that benefit from conversational answer generation. This avoids the operational complexity of replacing a working system while capturing the value of RAG for the use cases that justify it.

How Evaluation Differs

Traditional search evaluation uses information retrieval metrics: precision (fraction of retrieved documents that are relevant), recall (fraction of all relevant documents that were retrieved), MRR (Mean Reciprocal Rank - how high does the first relevant result appear), and NDCG (Normalized Discounted Cumulative Gain - a weighted measure of ranked list quality).

RAG evaluation requires a different framework because the output is a generated answer, not a ranked list. RAGAS measures faithfulness (whether generated claims are grounded in retrieved context), answer relevancy, context precision, and context recall. Traditional IR metrics apply to the retrieval component of a RAG pipeline. RAGAS metrics apply to the end-to-end pipeline.

A complete evaluation framework for a hybrid RAG system runs both sets of metrics. Retrieval metrics tell you whether the right documents are surfacing at the right ranks. RAGAS metrics tell you whether the end-to-end system is producing correct, grounded answers. Low context precision in RAGAS combined with good precision@10 in IR evaluation points to a reranking problem - the right documents are in the top 10 but are not being placed at the top of the context where the LLM can attend to them.

The Practical Decision

The decision between traditional search and RAG is not binary for most organizations. It is a question of what you add on top of what you already have, and where the generation layer earns its cost.

If users need documents, use traditional search. If users need answers synthesized from documents, use RAG. If your corpus mixes structured operational data with unstructured knowledge, use traditional search for the former and RAG for the latter.

If your queries require both exact terms and semantic matching, evaluate a combined retrieval layer. The complete BM25/vector fusion, RRF, implementation, and testing workflow belongs in Hybrid Search for RAG.

For how retrieval fits into the full RAG architecture, read RAG Architecture Explained. For vector storage, see Vector Database in RAG. For retrieval failures, see Why RAG Fails.

The final article in this series covers the embedding models that power both the dense retrieval side of hybrid search and the semantic understanding that makes RAG meaningfully better than pure keyword search: How Embeddings Work in RAG.

If you are still deciding whether RAG is the right architecture for your problem at all, start with What Is RAG in AI and RAG vs Fine-Tuning.

Related Reading

• What is RAG in AI? A simple explanation with examples