RAG Evaluation: Build a Reproducible Testing Pipeline

A single RAG score cannot tell you what broke. The retriever may miss the supporting document, or retrieval may succeed while the model produces an unsupported answer. Treating those outcomes as one number hides the component that needs work.

This guide builds a small evaluation pipeline that keeps the two layers separate. It uses a reviewed JSONL fixture, deterministic retrieval metrics, an optional Ragas run for generation quality, and a CI comparison against a recorded baseline. The core example runs with Python 3.11 and no API key.

Evaluate retrieval and generation independently, then combine the evidence in a release decision.

Define the Evaluation Contract First

Before choosing a library, define success for your application. A support assistant, legal search tool, and code assistant have different failure costs, so their release policies should differ.

Split the contract into two questions:

1. Did retrieval surface the evidence the answer needed? Compare ranked results with reviewed relevant document IDs.
2. Did generation use that evidence correctly? Evaluate support, task completion, refusal behavior, and citations as appropriate.

If the pipeline is still changing, start with RAG Architecture Explained. Every result should identify its retriever, reranker, prompt, model, corpus, and index versions.

Create a Small, Versioned Fixture

Start with the smallest reviewed set covering important query families and known failures. Three examples are enough to demonstrate the mechanics below; they are not enough to claim production readiness or statistical significance.

{"id":"refund-window","family":"policy","query":"How long do enterprise customers have to request a refund?","relevant_ids":["policy-refunds-v3"],"reference_answer":"Enterprise customers may request a refund within 60 days."}
{"id":"api-auth","family":"how-to","query":"How do I authenticate API v2 requests?","relevant_ids":["api-v2-auth","api-v2-quickstart"],"reference_answer":"Send a bearer token in the Authorization header."}
{"id":"unsupported-feature","family":"refusal","query":"Can the product export directly to COBOL?","relevant_ids":["supported-exports"],"reference_answer":"The documentation does not list COBOL as a supported export."}

Use stable IDs, label query families for coverage reporting, and record reviewed relevant document IDs. Add a corpus version or review date when sources change frequently.

Measure Retrieval Without an LLM Judge

Retrieval evaluation needs ranked IDs and relevance labels, so it can be fast and deterministic:

• Precision@k: the fraction of the first k results that are relevant.
• Recall@k: the fraction of labeled relevant documents found in the first k.
• Reciprocal rank: how early the first relevant result appears.

Save candidate output as candidate.jsonl:

{"id":"refund-window","retrieved_ids":["policy-refunds-v3","billing-faq","pricing"]}
{"id":"api-auth","retrieved_ids":["api-v2-quickstart","api-v2-auth","api-v1-auth"]}
{"id":"unsupported-feature","retrieved_ids":["supported-exports","roadmap","integrations"]}

Then compute metrics at an explicit cutoff:

# scripts/evaluate_retrieval.py
import json
import sys
from pathlib import Path

def read_jsonl(path):
    rows = {}
    for line in Path(path).read_text().splitlines():
        if line.strip():
            row = json.loads(line)
            if row["id"] in rows:
                raise SystemExit(f"Duplicate id: {row['id']}")
            rows[row["id"]] = row
    return rows

def score(relevant, retrieved, k):
    top_k = retrieved[:k]
    hits = sum(item in relevant for item in top_k)
    first_rank = next((rank for rank, item in enumerate(top_k, 1)
                       if item in relevant), None)
    return {
        f"precision@{k}": hits / k,
        f"recall@{k}": hits / len(relevant) if relevant else 1.0,
        f"mrr@{k}": 1 / first_rank if first_rank else 0.0,
    }

labels = read_jsonl(sys.argv[1])
runs = read_jsonl(sys.argv[2])
k = int(sys.argv[3])
if set(labels) != set(runs):
    raise SystemExit("Label and result IDs do not match")
per_case = [score(set(row["relevant_ids"]), runs[id]["retrieved_ids"], k)
            for id, row in labels.items()]
summary = {metric: round(sum(row[metric] for row in per_case) / len(per_case), 4)
           for metric in per_case[0]}
print(json.dumps({"cases": len(per_case), "metrics": summary}, indent=2))

python3.11 scripts/evaluate_retrieval.py eval/labels.jsonl eval/candidate.jsonl 3

The illustrative records produce:

{"cases": 3, "metrics": {"precision@3": 0.4444, "recall@3": 1.0, "mrr@3": 1.0}}

Every labeled document was found and the first relevant result ranked first, but most returned documents were not labeled relevant. Whether the extra context hurts depends on the generator and latency budget. For rank-order improvements, see RAG Reranking Explained.

Evaluate Generation Separately With Ragas

Retrieval metrics cannot determine whether prose is supported by context. Ragas offers model-assisted evaluation for dimensions such as groundedness, context relevance, and correctness. Its official workflow starts from an evaluation dataset and uses failures to guide iteration.

{
  "user_input": "How long is the enterprise refund window?",
  "retrieved_contexts": ["Enterprise customers may request a refund within 60 days."],
  "response": "Enterprise customers have 60 days to request a refund.",
  "reference": "Enterprise customers may request a refund within 60 days."
}

Follow the current Ragas documentation for installation and API syntax because integrations can change. Pin the validated version and record the judge model and prompt configuration.

Before using a model-judged score as a gate:

1. repeat the same sample to estimate normal variation;
2. manually review high-impact disagreements;
3. test clear-pass, clear-fail, and adversarial examples;
4. keep judge configuration constant between candidates;
5. report results per query family, not only as an aggregate.

The NVIDIA evaluation guide similarly separates answer accuracy, context relevance, groundedness, and retrieval recall. Select the dimensions your task requires: a research assistant needs citation checks, while a policy assistant may need strict abstention tests.

Add a Deterministic CI Regression Gate

Live judge calls introduce credentials, cost, latency, provider availability, and score variation into a merge path. Keep the required pull-request check deterministic: run the candidate retriever, calculate retrieval metrics, compare with a recorded baseline, and upload the full artifact.

# scripts/compare_eval.py
import json
import sys
from pathlib import Path

baseline = json.loads(Path(sys.argv[1]).read_text())
candidate = json.loads(Path(sys.argv[2]).read_text())
failures = []
if candidate["cases"] != baseline["cases"]:
    failures.append(f"case count: {baseline['cases']} -> {candidate['cases']}")
for metric, old in baseline["metrics"].items():
    new = candidate["metrics"][metric]
    allowed = baseline["allowed_drop"].get(metric, 0.0)
    if old - new > allowed:
        failures.append(f"{metric}: {old:.4f} -> {new:.4f}; allowed {allowed:.4f}")
if failures:
    raise SystemExit("\n".join(failures))
print("RAG retrieval regression check passed")

{
  "cases": 3,
  "metrics": {"precision@3": 0.4444, "recall@3": 1.0, "mrr@3": 1.0},
  "allowed_drop": {"precision@3": 0.0, "recall@3": 0.0, "mrr@3": 0.0}
}

Zero tolerance fits this tiny fixture because one case creates a large jump. It is not a universal recommendation. Set production tolerances after repeated runs and according to application risk.

name: RAG evaluation
on:
  pull_request:
    paths: ["src/retrieval/**", "config/retrieval/**", "eval/**", "scripts/**"]
jobs:
  retrieval-regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.11"}
      - run: python3 scripts/run_retriever_fixture.py > eval/candidate.jsonl
      - run: python3 scripts/evaluate_retrieval.py eval/labels.jsonl eval/candidate.jsonl 3 > eval/result.json
      - run: python3 scripts/compare_eval.py eval/baseline.json eval/result.json

run_retriever_fixture.py is the adapter to your retriever. It must emit one stable ID and ranked list for every label. Run slower model-judged checks separately for prompt, model, index, or release changes unless your team has explicitly designed them as a merge gate.

Feed Reviewed Production Failures Back Into Tests

Pre-launch examples cannot anticipate every production query. A safe feedback loop is:

1. capture query, retrieved IDs, answer, system versions, latency, and user signal;
2. redact secrets and personal data before evaluation storage;
3. place failures in a human review queue;
4. identify correct supporting documents and expected behavior;
5. add the reviewed case with a stable ID and query family;
6. reproduce the failure before changing the system;
7. retain the case after the fix.

The NVIDIA observability documentation describes traces and metrics across RAG services. Collection alone is not evaluation: reviewed expectations and an explicit release policy are still required.

Avoid feedback bias. Thumbs-down cases overrepresent visible failures, while quietly incomplete answers may receive no signal. Subject to privacy rules, review a sample of ordinary traffic too.

Keep Quality and Performance Evidence Together

A candidate can retrieve better but become too slow or expensive. Record latency and resource measurements beside quality results without presenting them as the same thing:

• retrieval and end-to-end latency percentiles;
• cutoff k and reranker depth;
• prompt and response token counts;
• model, embedding, index, and corpus versions;
• errors, timeouts, and empty-retrieval rates.

Benchmark representative hardware and concurrency. The NVIDIA performance guide distinguishes pipeline latency and throughput measurements; these operational results are not evidence of answer quality.

Common Evaluation Mistakes

• Using one aggregate score. Preserve layer-level and query-family results so one improvement cannot hide another regression.
• Copying thresholds. A score has no universal meaning across datasets, judges, prompts, and risks. Establish and document your own baseline.
• Changing everything together. Version labels, corpus, prompts, models, and indexes so a score movement remains explainable.
• Overclaiming from a tiny fixture. A demo proves the pipeline runs; it does not estimate all production traffic.
• Automatically promoting feedback. Dissatisfaction is not a verified label. Require review before a trace becomes ground truth.

Release Checklist

• The fixture covers affected query families and known failures.
• Labels and source documents are reviewed and versioned.
• Every case produces exactly one result.
• Candidate and baseline use the same metric definitions and cutoff.
• Judge configuration is recorded for generation checks.
• Results are reviewed per family, not only in aggregate.
• Quality, latency, and errors are reported separately.
• The artifact identifies code, corpus, index, prompt, and model versions.

When retrieval fails, investigate chunking, filtering, or ranking. Why RAG Fails covers those patterns. When generation fails despite good evidence, investigate context formatting, instructions, citations, and refusal behavior.

The goal is not a perfect dashboard number. It is a repeatable comparison showing what changed, where it changed, and whether the evidence is strong enough to release.

Related Reading

• RAG Architecture Explained
• Why RAG Fails
• RAG Reranking Explained