RAG Reranking Explained: Models, Metrics, and Production Trade-Offs

RAG reranking is a second-stage retrieval step that reorders a small candidate set before the language model receives context.

The basic pipeline is:

query -> fast retrieval (top 50) -> reranker (top 5) -> LLM

The first retriever searches the whole corpus efficiently. The reranker spends more computation comparing the query with only the retrieved candidates. Its job is not to discover new documents. Its job is to move the most useful evidence upward and push plausible-but-wrong chunks out of the final prompt.

This distinction matters. If the answer-bearing chunk is missing from the first 50 candidates, reranking cannot rescue it. If that chunk is present at rank 18 but the application sends only the first five chunks to the LLM, reranking may be exactly the right fix.

This guide explains how reranking works, which model families to consider, how it fits after vector or hybrid search, and how to determine whether it improves your system. For the complete surrounding pipeline, begin with the production RAG architecture guide.

What Is Reranking in RAG?

Reranking means scoring an existing list of retrieved chunks again with a model designed for finer relevance judgments.

Suppose vector search returns these results for the query Can contractors export customer data?:

1. How customers export account data
2. Contractor access-control policy
3. Data retention for former contractors
4. Export restrictions for external contractors  <- answer
5. Customer data-processing agreement

The list is semantically reasonable, but the best evidence is fourth. A reranker reads the query with each candidate and may produce:

1. Export restrictions for external contractors  <- answer
2. Contractor access-control policy
3. How customers export account data
4. Customer data-processing agreement
5. Data retention for former contractors

The candidate set has not changed. The ordering has.

This two-stage pattern exists because retrieval systems have competing goals:

• First-stage retrieval optimizes coverage and speed. It must search thousands or millions of chunks.
• Reranking optimizes precision near the top. It can use a slower model because it sees only tens of candidates.
• Context selection optimizes the final evidence set. It limits prompt noise, duplication, latency, and token cost.

Sentence Transformers' retrieve-and-rerank documentation describes the same pattern: an efficient retriever generates candidates, then a cross-encoder improves their final ranking.

Retrieval Failure or Ranking Failure?

Before adding a reranker, identify which failure you have.

Retrieval failure

The required evidence is not present in the candidate set.

Relevant chunk rank: not in top 50
Recall@50: failed

A reranker cannot score a chunk it never receives. Improve:

• Chunk boundaries and document parsing
• Query rewriting or expansion
• Embedding model and index settings
• Metadata filters
• Candidate depth
• Keyword coverage with hybrid search

Ranking failure

The evidence is present, but it appears below irrelevant or less useful chunks.

Relevant chunk rank before reranking: 17
Relevant chunk rank after reranking: 2
Recall@50: unchanged
MRR and Precision@5: improved

That is the problem reranking is built to solve.

How a Cross-Encoder Reranker Works

Dense retrieval normally uses a bi-encoder. It encodes the query and document separately:

query -> query embedding ----\
                              cosine similarity
chunk -> chunk embedding ----/

Because document embeddings can be computed once and indexed, this design scales well. The trade-off is that the model does not jointly examine every query token against every document token at retrieval time.

A cross-encoder instead processes the pair together:

[query tokens] [SEP] [candidate tokens] -> transformer -> relevance score

Joint attention can distinguish details such as negation, conditions, entity relationships, versions, and which passage directly answers the question. It is more expensive because the model must run once for every query-candidate pair.

BAAI's BGE reranker model card describes this trade-off directly: full attention over the input pair is more accurate but more time-consuming than an embedding model, so the cross-encoder reranks only the top candidates.

For a deeper architectural comparison, read dual encoders vs cross-encoders. The practical lesson for RAG is simple: retrieve broadly with the cheaper model, then judge narrowly with the stronger one.

Reranker Types for RAG

Cross-encoders are common, but they are not the only option.

Cross-encoder rerankers

A cross-encoder produces a relevance score for each query-document pair. Examples include BGE rerankers and hosted reranking APIs.

Use a cross-encoder when you need:

• Strong pointwise relevance scoring
• A straightforward top-k reranking stage
• Self-hosted or managed deployment options
• Predictable batching over a limited candidate set

Its main costs are inference latency and maximum input length. Long chunks may be truncated, so verify what portion of each document the model actually reads.

Late-interaction rerankers

Late-interaction systems such as ColBERT keep multiple token-level representations instead of reducing each document to a single vector. They preserve more fine-grained matching while allowing document representations to be precomputed.

They can provide a useful middle ground between single-vector retrieval and full cross-encoding, but they introduce larger indexes and more involved serving infrastructure.

LLM rerankers

An LLM can score candidates individually, compare a list, or select passages that contain enough evidence to answer the query. This allows instructions such as:

Prefer passages that state the effective policy directly.
Reject documents describing an older product version.
Return diverse evidence for every part of the question.

LLM rerankers are flexible but tend to be slower, more expensive, and less deterministic. Position bias can affect listwise ranking, and pairwise comparisons can require many calls. Reserve them for workloads where instruction-following or answer usefulness matters enough to justify the cost.

Rule-based and business reranking

Production rankings often combine semantic relevance with deterministic rules:

• Exclude documents outside the user's permissions
• Prefer the current policy over an archived version
• Boost the requested language or product
• Demote duplicate passages
• Preserve evidence from different parent documents

Hard access-control and tenant filters should run before reranking. A reranker must never receive content the user is not authorized to see.

Where Reranking Fits After Hybrid Search

Hybrid search and reranking solve different problems.

Hybrid search combines complementary ways to find candidates:

BM25 top 50 -------\
                    RRF top 60 -> reranker top 6 -> LLM
vector top 50 -----/

BM25 protects exact terms such as identifiers, error codes, API names, and versions. Dense retrieval finds paraphrases and conceptual matches. Reciprocal Rank Fusion combines the two rankings without pretending their raw scores share a scale. The reranker then performs a deeper query-candidate comparison.

Fusion improves candidate coverage. Reranking improves final order. Learn the complete fusion workflow in Hybrid Search for RAG Explained.

Candidate Depth and Final Top-K

Two numbers control the reranking stage:

• Candidate depth: how many retrieved chunks the reranker receives
• Final top-k: how many reranked chunks survive for context selection

These values should not be identical. Retrieving only the five chunks you plan to send to the LLM gives the reranker almost no opportunity to correct the first-stage order.

A practical evaluation grid is:

Do not treat 40 and 5 as universal defaults. Test them against your corpus and traffic.

Candidate depth creates a quality-cost curve:

too shallow -> relevant evidence excluded before reranking
large enough -> evidence included and reordered successfully
too deep -> higher latency, cost, noise, and truncation risk

Measure Recall@candidate-k before reranking. If recall stops improving between 40 and 80 candidates, the larger window may not justify its cost. If recall continues to improve but reranking latency becomes unacceptable, improve first-stage retrieval so relevant evidence appears earlier.

Truncation: the Hidden Reranking Failure

Rerankers have input limits. When the query and candidate exceed that limit, the implementation may truncate part of the candidate.

Imagine a long chunk structured as:

Title and navigation
General policy introduction
Definitions
Exceptions
Answer-bearing sentence near the end  <- truncated

The model cannot rank evidence it never reads. This often looks like a model-quality problem when it is really an ingestion or input-format problem.

To reduce truncation failures:

• Use coherent chunks sized for both embedding and reranking models
• Put titles, section headings, and important metadata next to the chunk text
• Remove navigation, boilerplate, and repeated footers
• Log token counts and truncation rates
• Test queries whose answer appears near chunk boundaries
• Use a long-context reranker only if evaluation shows the extra context helps

Read how to choose chunk size for RAG before compensating for broken chunks with a larger model.

Deduplication and Evidence Diversity

A relevance reranker may place four overlapping chunks from one document in the top five. Each chunk is individually relevant, but together they waste context and can hide another source needed for the answer.

Use a post-reranking selection stage:

def select_diverse(results, limit=5, max_per_document=2):
    selected = []
    per_document = {}

    for result in results:
        document_id = result["document_id"]
        count = per_document.get(document_id, 0)

        if count >= max_per_document:
            continue

        if overlaps_selected_evidence(result, selected):
            continue

        selected.append(result)
        per_document[document_id] = count + 1

        if len(selected) == limit:
            break

    return selected

For single-fact questions, one highly relevant passage may be enough. For comparisons and multi-hop questions, diversity across entities or documents becomes part of relevance. Evaluate these query types separately.

Reranking Scores and Thresholds

Reranking scores are model outputs, not universal probabilities.

A score of 0.7 from one model is not necessarily comparable to 0.7 from another model, another version, or another domain. Even within one model, the useful threshold depends on your candidates and tolerance for unsupported answers.

Use scores in three ways:

1. Ordering: sort candidates by score.
2. Filtering: remove weak candidates using a threshold calibrated on labeled queries.
3. Abstention: if no candidate passes the threshold, tell the generation layer that the corpus may not contain an answer.

Do not copy a threshold from a tutorial. Plot score distributions for relevant and irrelevant candidates in your own evaluation set. Recalibrate whenever you change the reranker, chunking, corpus, or first-stage retriever.

How to Evaluate a RAG Reranker

Evaluate retrieval and generation separately. Otherwise, an LLM may produce a plausible answer from prior knowledge and hide a weak retrieval pipeline.

Build a labeled query set

For every query, record:

• Relevant chunk IDs or documents
• The minimum evidence needed to answer
• Query type: factual, identifier, comparison, multi-hop, or unanswerable
• Required access-control and metadata filters
• Expected answer and supporting citations

Include hard negatives: chunks that share the vocabulary and topic but do not answer the question. These examples reveal whether the reranker understands fine distinctions rather than broad similarity.

Run an ablation

Keep the corpus, queries, filters, and generation settings fixed.

Important metrics include:

• Recall@candidate-k: did the retriever supply the required evidence?
• Mean Reciprocal Rank: how early does the first relevant result appear?
• nDCG@k: does the top of the list contain highly relevant results in the right order?
• Precision@k or context precision: how much selected context is useful?
• Duplicate rate: how much context repeats the same evidence?
• Answer faithfulness and citation accuracy: does generation remain grounded?
• Latency, throughput, and cost: is the improvement operationally acceptable?

Sentence Transformers provides a CrossEncoderRerankingEvaluator with reranking metrics including MRR. For an end-to-end testing strategy, continue with RAG evaluation engineering.

Read the metrics in the correct order

If Recall@50 is low, fix retrieval first.

If Recall@50 is good but MRR, nDCG@5, or context precision is weak, test a reranker.

If retrieval metrics improve but answer quality does not, inspect prompt construction, context order, conflicting evidence, and the generator. The bottleneck has moved downstream.

A Minimal Cross-Encoder Implementation

The exact model is less important than preserving candidate IDs and measuring the change.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base")

def rerank(query, candidates, final_k=5):
    pairs = [(query, item["text"]) for item in candidates]
    scores = reranker.predict(pairs, batch_size=32)

    ranked = [
        {**candidate, "rerank_score": float(score)}
        for candidate, score in zip(candidates, scores)
    ]

    ranked.sort(key=lambda item: item["rerank_score"], reverse=True)
    return ranked[:final_k]

In production, add:

• Batched inference
• Maximum-length monitoring
• Candidate and score logging with privacy controls
• Timeouts and a retriever-order fallback
• Model-version tracking
• Deduplication and diversity selection
• A frozen evaluation set for every deployment

BAAI recommends using its cross-encoder rerankers on the top-k documents returned by an embedding model. The BGE model documentation also provides self-hosted usage paths and model variants.

Hosted or Self-Hosted Reranking?

Choose deployment based on data policy, traffic, and operating capability.

Hosted reranking is useful for quickly establishing whether the stage improves quality. Self-hosting becomes attractive when privacy, predictable high volume, offline operation, or model customization dominates.

Benchmark with your candidate lengths and concurrency. A model that looks fast on ten short passages may behave differently on 80 long technical chunks.

Production Failure Modes

Reranking too few candidates

If the relevant result is usually ranked 30 to 50, reranking only the first ten cannot help. Measure candidate recall at several depths.

Sending every high-scoring chunk to the LLM

More relevant text is not automatically better context. Control duplicates, contradictions, and total tokens.

Applying permissions after reranking

Authorization filters must run before any model sees candidates. Post-filtering can also leave too few results and distort evaluation.

Treating relevance as answer sufficiency

A chunk can be topically relevant without containing the fact needed to answer. Label answer-bearing evidence, not merely related documents.

Ignoring document freshness

A reranker may prefer an older policy because its wording matches the query more closely. Apply effective-date rules or incorporate freshness explicitly.

Changing the retriever without retesting the reranker

The first-stage retriever determines the reranker's candidate distribution. Switching embeddings, hybrid weights, or chunking changes the hard negatives it sees.

No timeout fallback

A remote reranking failure should not necessarily fail the entire request. Define whether the system returns first-stage results, retries, or abstains.

When Reranking Is Worth It

Reranking is a strong candidate when:

• Relevant chunks often appear in the candidate set but below the final cutoff
• The corpus contains many near-duplicate or closely related passages
• Questions depend on negation, conditions, relationships, or versions
• Incorrect context has a high product or business cost
• Hybrid retrieval improves recall but produces a noisy fused order

Skip or postpone it when:

• The corpus is small enough for a simpler search method
• The top results are already precise on representative queries
• Latency is strict and reranking provides no measured downstream gain
• Candidate recall is poor
• Broken parsing or chunking is the actual bottleneck

For a wider failure diagnosis, see why RAG systems fail.

A Practical Starting Experiment

Use this sequence as an experiment, not a universal configuration:

1. Freeze a representative labeled query set.
2. Record the retriever-only baseline.
3. Retrieve 40 candidates per query.
4. Rerank those candidates with one cross-encoder.
5. Select the top 5 after deduplication.
6. Compare Recall@40, MRR, nDCG@5, context precision, and latency.
7. Repeat with 20 and 80 candidates.
8. Test final context sizes of 3, 5, and 8.
9. Evaluate answer faithfulness and citation accuracy.
10. Ship only if the held-out improvement justifies the cost.

Then analyze results by query type. Reranking may create a large improvement for policy exceptions and version-specific questions while doing little for direct keyword lookups. That breakdown is more useful than one overall average.

The Bottom Line

RAG reranking is not another retrieval system. It is a precision layer between broad candidate retrieval and narrow context selection.

The durable rules are:

• A reranker cannot recover missing candidates
• Cross-encoders trade speed for deeper query-document interaction
• Candidate depth and final top-k solve different problems
• Truncation and duplicate evidence can erase the expected gain
• Scores and thresholds must be calibrated on your data
• Retrieval metrics should improve before you credit reranking for better answers

Start by checking candidate recall. If the evidence is present but poorly ordered, reranking is a focused, testable improvement. If the evidence is absent, repair retrieval first.

Next, place reranking inside the full RAG architecture, or combine it with hybrid search for RAG when your queries need both exact and semantic matching.