Hybrid Search for RAG Explained: BM25, Vector Search, and RRF

Hybrid search for RAG combines keyword retrieval and semantic retrieval before the language model sees any context.

The practical version usually looks like this:

User query
   |-- BM25 keyword search ------> ranked list A
   |-- dense vector search -----> ranked list B
                                      |
                              fusion (often RRF)
                                      |
                              optional reranker
                                      |
                              top chunks for the LLM

The reason is simple. Dense vector search is good at meaning, paraphrases, and natural-language questions. BM25 is good at literal terms, rare words, identifiers, model names, error codes, and numbers. Real users ask both kinds of questions, often in the same query.

Hybrid search is not a replacement for good chunking, filters, or evaluation. It is a retrieval strategy that gives your RAG pipeline two different ways to find evidence. If you need the complete system around it, start with the pillar guide to production RAG architecture. If vector retrieval itself is new, read what vector search is first.

What Is Hybrid Search for RAG?

Hybrid search is a retrieval method that runs two or more complementary search systems and combines their results into one ranking.

In a typical RAG system, the two retrievers are:

• A lexical retriever, usually BM25, that scores exact term overlap
• A dense retriever that compares embedding vectors for semantic similarity

Each retriever returns a list of document chunks. A fusion method merges those lists. The final candidates may go directly into the prompt, but production systems often add a reranker before selecting the last three to ten chunks.

The word hybrid can describe other combinations—multiple embedding models, graph and text retrieval, or dense plus learned sparse vectors. In this guide, it means the most common RAG pattern: BM25 or sparse lexical retrieval plus dense vector retrieval.

Pinecone's hybrid-search documentation describes the same motivation: semantic search can miss exact domain terminology, while lexical search can miss synonyms and paraphrases. Qdrant's documentation uses the example of combining conceptual queries with exact book identifiers. Both are retrieval problems that appear constantly in enterprise RAG.

Why Dense Vector Search Alone Misses Relevant Chunks

Dense retrieval converts the query and every chunk into embeddings. It then finds chunks whose vectors are close to the query vector.

That works well for semantic equivalence:

Query:    "How can I get my money back?"
Document: "Refund requests are accepted within 30 days."

The words differ, but the meanings align.

The weakness appears when a query depends on exact surface forms:

ERR_AUTH_1042
PostgreSQL 42P01
invoice INV-2026-00817
model XG-14B-v2
section 7.3(b)

An embedding model may understand that ERR_AUTH_1042 concerns authentication, but the user does not want a generally relevant authentication page. They want the chunk containing that exact code.

Dense search can also blur distinctions between nearby concepts:

• Version 2.1 versus version 2.2
• Starter plan versus Standard plan
• Read timeout versus connection timeout
• delete_user versus disable_user

Those items are semantically close. Their operational meaning is different.

This does not make embeddings unreliable. It means semantic similarity is not the only relevance signal in technical documents, policies, catalogs, support tickets, source code, and regulated records.

Why BM25 Alone Also Fails

BM25 ranks documents using term occurrence, term rarity, and document-length normalization. It is a strong lexical baseline because rare query terms receive more weight than common terms, repeated occurrences have diminishing returns, and long documents are normalized rather than rewarded automatically.

BM25 performs well when the important words are present in both the query and the document:

Query:    "ERR_AUTH_1042"
Document: "ERR_AUTH_1042 occurs when an access token has expired."

But it cannot reliably infer that different expressions mean the same thing:

Query:    "The app forgets who I am after a few minutes."
Document: "Session tokens expire after 15 minutes of inactivity."

Tokenization, stemming, synonyms, and query expansion can improve lexical retrieval. They do not give BM25 the broad semantic representation that dense embeddings provide.

The two retrievers therefore fail differently:

Hybrid retrieval is useful because the errors are complementary.

How the Hybrid RAG Pipeline Works

Hybrid retrieval has five operational steps.

1. Index the same chunks for both retrieval paths

Every chunk needs a stable identifier shared across the lexical and dense indexes.

chunk_id: policy-2026-refunds-04
text: "Enterprise customers may request a refund within 45 days..."
metadata:
  product: enterprise
  document_type: policy
  effective_date: 2026-04-01

The dense index stores the chunk's embedding. The lexical index stores its searchable text and term statistics. Some systems keep both representations in one collection; others use separate services. Stable IDs are what make fusion and deduplication possible.

2. Run both retrievers for the same query

The dense path embeds the query and performs nearest-neighbor search. The lexical path tokenizes the query and runs BM25 or a learned sparse model.

Query: "What does ERR_AUTH_1042 mean after token refresh?"

Dense path finds:
1. Token refresh troubleshooting
2. Authentication lifecycle
3. Expired sessions

BM25 path finds:
1. ERR_AUTH_1042 reference
2. Authentication error catalog
3. Token refresh troubleshooting

Neither list is obviously complete on its own.

3. Apply the same hard filters

Permissions, tenant boundaries, language, product, document status, and effective dates should be applied consistently to both paths.

If BM25 searches all documents while the vector query filters to one tenant, fusion can surface content the user should never receive. Access control belongs inside retrieval, not after the LLM has already seen the text.

4. Fuse the ranked lists

Raw BM25 scores and vector similarity scores are not directly comparable. BM25 scores are unbounded and change with query terms and corpus statistics. Cosine similarity usually lives in a much smaller range.

Adding the raw scores is therefore a mistake:

# Do not do this without normalization.
hybrid_score = bm25_score + cosine_similarity

Use a rank-based method such as Reciprocal Rank Fusion, or normalize each score distribution before using a weighted combination.

5. Rerank and select context

Fusion improves candidate recall. It does not deeply compare the query with every candidate.

A cross-encoder or late-interaction reranker can take the fused top 20 to 100 candidates, score them more precisely, and return the final chunks. Qdrant's hybrid search with reranking tutorial demonstrates this dense-plus-sparse retrieval followed by a late-interaction stage.

The final prompt should contain only the evidence the model needs, not every candidate returned by both retrievers.

Reciprocal Rank Fusion Explained

Reciprocal Rank Fusion, or RRF, combines ranked lists without using their raw scores.

For a document d, the common form is:

RRF(d) = sum(1 / (k + rank_r(d)))

The sum covers every retriever in which the document appears. k is a constant that controls how quickly rank contributions decay. The original 2009 RRF paper used k = 60 and found that the simple fusion method consistently outperformed the individual systems in its experiments. You can read the original University of Waterloo paper.

Suppose the rankings are:

With k = 60:

Exact error reference = 1/61 + 1/66 = 0.03154
Token refresh guide   = 1/63 + 1/61 = 0.03227
Authentication page   = 1/68 + 1/62 = 0.03084

The token refresh guide wins because it performs well in both retrieval systems. The exact error reference remains competitive because it ranks first in BM25.

RRF has four practical advantages:

• It avoids incompatible score scales
• It works when a document appears in only one list
• It requires little tuning to establish a baseline
• It is easy to inspect during retrieval debugging

Azure AI Search, Elasticsearch, and Qdrant all support rank-based fusion in their hybrid-search workflows.

A Minimal RRF Implementation in Python

The fusion logic is small enough to own in application code:

from collections import defaultdict
from dataclasses import dataclass


@dataclass(frozen=True)
class SearchResult:
    chunk_id: str
    text: str
    score: float


def reciprocal_rank_fusion(
    ranked_lists: list[list[SearchResult]],
    rank_constant: int = 60,
) -> list[tuple[SearchResult, float]]:
    fused_scores: dict[str, float] = defaultdict(float)
    results_by_id: dict[str, SearchResult] = {}

    for results in ranked_lists:
        for rank, result in enumerate(results, start=1):
            results_by_id[result.chunk_id] = result
            fused_scores[result.chunk_id] += 1 / (rank_constant + rank)

    ranked_ids = sorted(
        fused_scores,
        key=fused_scores.get,
        reverse=True,
    )

    return [
        (results_by_id[chunk_id], fused_scores[chunk_id])
        for chunk_id in ranked_ids
    ]


bm25_results = [
    SearchResult("error-code", "ERR_AUTH_1042 means...", 12.4),
    SearchResult("error-catalog", "Authentication errors...", 8.1),
    SearchResult("refresh-guide", "Refresh access tokens...", 6.7),
]

dense_results = [
    SearchResult("refresh-guide", "Refresh access tokens...", 0.89),
    SearchResult("auth-lifecycle", "Authentication lifecycle...", 0.86),
    SearchResult("error-code", "ERR_AUTH_1042 means...", 0.79),
]

for result, rrf_score in reciprocal_rank_fusion(
    [bm25_results, dense_results]
):
    print(result.chunk_id, round(rrf_score, 5))

Notice that the function ignores 12.4 versus 0.89. Those values are meaningful within their own retrievers, but not across retrievers.

In production, add:

• A per-retriever candidate limit
• Optional retriever weights validated on an evaluation set
• Stable tie-breaking
• Deduplication by chunk and parent document
• Retrieval traces that preserve each original rank and score

RRF vs Weighted Score Fusion

RRF is not the only fusion method.

RRF

RRF uses rank positions. It is the safest starting point when you do not have labeled relevance judgments or trustworthy score calibration.

Its limitation is that it discards score magnitude. A vector result that is barely first and one that is decisively first receive the same rank contribution.

Normalized weighted fusion

Weighted fusion normalizes BM25 and vector scores, then blends them:

hybrid_score = alpha * normalized_dense_score
             + (1 - alpha) * normalized_lexical_score

This gives you control over how much each retriever contributes. It also creates more parameters to tune and can become unstable if score distributions shift between queries.

Weaviate's hybrid API exposes an alpha value, where 0 is pure keyword retrieval and 1 is pure vector retrieval. Its current default fusion method uses normalized relative scores. Pinecone documents an explicit weighting pattern because its sparse and dense values do not naturally share a scale.

Use this decision rule:

Do not choose based on a generic benchmark. Choose using your queries and relevance judgments.

Hybrid Search Is Not Reranking

These stages solve different problems.

Hybrid retrieval increases coverage. BM25 and vector search provide a broad candidate set through complementary signals.

Fusion combines candidate rankings. RRF or normalized scoring creates one list.

Reranking improves ordering. A cross-encoder reads the query and each candidate together to estimate relevance more precisely.

BM25 top 50 -----\
                  > RRF top 80 -> reranker top 8 -> LLM
Vector top 50 ---/

Why retrieve 50 from each path if the LLM needs only five chunks? Because the first-stage goal is recall. If the answer-bearing chunk never enters the candidate pool, no reranker or language model can recover it.

Candidate depth still needs measurement. Retrieving hundreds of candidates increases latency and reranking cost. Retrieving only five from each path can prevent fusion from finding useful overlap.

A reasonable experiment is:

BM25 candidates: 20, 50, 100
Dense candidates: 20, 50, 100
Fused candidates sent to reranker: 20, 40, 80
Final context chunks: 3, 5, 8

Measure recall and latency at every step.

When Hybrid Search Helps Most

Hybrid retrieval is especially useful for:

Developer documentation

Queries mix intent with exact API methods, exceptions, package names, and versions.

"Why does Prisma P2025 happen after updateMany?"

Dense retrieval understands the failure scenario. BM25 protects P2025 and updateMany.

Customer support knowledge bases

Users describe problems conversationally but also paste exact error text, product names, invoice IDs, or plan names.

Legal and policy search

Semantic retrieval finds conceptually related clauses. Lexical retrieval protects section numbers, defined terms, dates, and jurisdiction-specific phrases.

Product catalogs

Natural-language intent must coexist with SKUs, brands, materials, dimensions, and model numbers.

Scientific and medical corpora

Synonyms and conceptual similarity matter, but so do exact chemical names, gene symbols, study identifiers, and abbreviations.

When Hybrid Search May Not Improve RAG

Hybrid is not an automatic upgrade.

It may add little value when:

• Queries are almost entirely conversational and semantic
• The corpus contains few exact identifiers or rare terms
• A domain-specific dense model already retrieves nearly every relevant chunk
• The knowledge base is so small that exhaustive or full-context retrieval works
• Lexical tokenization performs poorly for the language or document format
• Chunk boundaries are broken, so neither retriever has a complete answer-bearing unit

Hybrid search can also reduce quality when a weak lexical retriever introduces noisy chunks and fusion gives it too much influence.

That is why “hybrid beats dense” is not a production conclusion. It is a hypothesis to test.

Common Hybrid Search Mistakes

Adding raw scores together

BM25 and cosine scores do not share a meaningful scale. Use RRF or normalize them first.

Retrieving too few candidates

Fusion cannot recover documents excluded before fusion. Evaluate the candidate window, not only the final top-k.

Using different filters on each retriever

Tenant, permissions, language, status, and date rules must match across retrieval paths.

Confusing duplicate chunks with agreement

Overlapping chunks from the same document can occupy several top positions. Deduplicate by chunk and consider parent-document diversity before sending context to the LLM.

Tuning alpha without a labeled set

An alpha of 0.7 is not inherently better than 0.5. A weight is a model parameter. Tune it on held-out queries.

Evaluating only final answers

A language model can answer correctly despite weak retrieval by relying on prior knowledge. Inspect retrieval metrics separately so a lucky answer does not hide a broken retriever.

Assuming hybrid fixes ingestion

No fusion method can reconstruct a table split across unrelated chunks or recover metadata that was never indexed. Fix chunk size and boundaries first.

How to Evaluate Hybrid Search for Your RAG System

Create a small golden dataset before tuning.

For each query, record:

• The relevant chunk IDs or documents
• The query type
• Required filters
• Whether one chunk or multiple chunks are needed
• The expected answer, if evaluating generation too

Your query groups should include:

semantic:     "How can I stop users losing their sessions?"
identifier:   "ERR_AUTH_1042"
mixed:        "What causes ERR_AUTH_1042 after refresh?"
numeric:      "What changed between API v2.1 and v2.2?"
multi-hop:    "Compare enterprise and starter refund limits"
negative:     "A question the corpus cannot answer"

Run an ablation table:

The important metrics are:

• Recall@k: whether the candidate set contains the required evidence
• Hit rate: whether at least one relevant chunk appears in the top-k
• Mean reciprocal rank: how early the first relevant chunk appears
• Context precision: how much retrieved context is useful rather than noise
• Duplicate rate: how many top results repeat the same evidence
• Latency and cost: the operational price of the improvement
• Answer faithfulness: whether the generated answer stays supported by retrieved evidence

Break results down by query type. A single average can hide the actual benefit: dense search may already dominate semantic questions while hybrid retrieval produces a large improvement only for identifier-heavy queries. That can still justify hybrid search if those queries are important.

For a complete evaluation pipeline and regression testing, continue with RAG evaluation engineering.

A Practical Production Starting Point

Use this as a baseline, not a universal optimum:

1. Index the same structured chunks for lexical and dense retrieval.
2. Apply identical permissions and metadata filters to both paths.
3. Retrieve 40 candidates from BM25 and 40 from vector search.
4. Fuse with RRF using the engine's documented default.
5. Deduplicate and preserve source diversity.
6. Rerank the top 30 to 50 candidates when precision matters.
7. Send the best 3 to 8 chunks to the LLM.
8. Compare against dense-only and BM25-only baselines.

Do not copy 40, 50, or 8 into production without testing. These values create a useful experiment because they separate candidate recall from final context precision.

Start with RRF if you lack relevance labels. Once you have enough evaluation data, test weighted RRF or normalized score fusion. Keep the simpler method unless the tuned alternative produces a repeatable improvement on held-out queries.

The Bottom Line

Hybrid search improves RAG by giving retrieval two complementary signals:

• BM25 protects exact terms and identifiers
• Dense vectors capture meaning and paraphrases
• Fusion combines their candidate rankings
• Reranking sharpens the final order
• Evaluation determines whether the extra path is worth operating

The most important lesson is not “always use hybrid search.” It is measure retrieval by query type. If vector search misses codes and BM25 misses intent, hybrid retrieval is a strong solution. If dense retrieval already solves your workload, extra machinery may only add latency and debugging surface.

Next, place hybrid retrieval in the wider production RAG architecture, then use RAG evaluation engineering to test it against a frozen query set.