RAG Architecture Explained: How Production Pipelines Actually Work (2026)
A practical guide to production RAG architecture, including chunking, embeddings, hybrid search, reranking, query transformation, agentic RAG, evaluation, and the trade-offs between each layer.
The tutorial version of RAG is three steps. Index documents. Retrieve chunks. Generate an answer. Thirty lines of Python. Works in a demo.
Then you put it in front of real users, with real documents, asking real questions. And retrieval starts returning the wrong chunks. The model answers confidently. The answers are wrong. You look at the retrieved context and you cannot even tell why those chunks ranked highest.
That is not bad luck. That is what happens when you build naive RAG and call it done.
Retrieval failures are easy to hide because the generation layer can turn incomplete context into a fluent answer. Fixing that problem requires inspecting every component between the source documents and the evidence passed to the model.
This article covers all of them. New to vector databases? The complete beginner guide to vector databases covers the fundamentals.
The Full Production Architecture
A production RAG pipeline separates ingestion, retrieval, reranking, and generation so each stage can be evaluated.
Before going layer by layer, here is the complete picture of what a production RAG system looks like in 2026.
INGESTION PIPELINE (runs offline)
+------------------------------------------------------------------+
| |
| Raw Documents |
| (PDF, DOCX, HTML, DB records, Markdown) |
| | |
| v |
| Document Parser |
| (extract text, structure, and source metadata) |
| | |
| v |
| Chunking Layer |
| (create retrievable units with stable IDs) |
| | |
| v |
| Embedding Layer |
| (apply one versioned representation contract) |
| | |
| v |
| Retrieval Store + Metadata |
| (persist vectors, payloads, and source links) |
| |
+------------------------------------------------------------------+
QUERY PIPELINE (runs at inference time)
+------------------------------------------------------------------+
| |
| User Question |
| | |
| v |
| Query Transformation (optional) |
| (prepare one or more retrieval requests) |
| | |
| v |
| Retrieval |
| (one or more search paths) |
| | |
| v |
| Candidate Merge |
| (produce one traceable result set) |
| | |
| v |
| Candidate Ordering (optional) |
| (reduce and reorder evidence) |
| | |
| v |
| Context Assembly |
| (top-k chunks + metadata + citation markers) |
| | |
| v |
| LLM Generation |
| (answer from the assembled evidence) |
| | |
| v |
| Answer with Citations |
| |
+------------------------------------------------------------------+
EVALUATION LAYER (runs continuously)
+------------------------------------------------------------------+
| Evaluate retrieval and generation separately |
| Preserve traces across every component boundary |
| Monitor quality, latency, cost, and regressions |
+------------------------------------------------------------------+A production RAG pipeline combines ingestion, representation, retrieval, context assembly, generation, and evaluation. Keep their contracts explicit and components replaceable so evidence from evaluation can drive changes without forcing a full-system rewrite.
Now, each layer in detail.
Layer 1: Document Ingestion and Parsing
Before you can chunk anything, you need clean text. This is the step most tutorials skip, and it creates problems that cascade through every layer downstream.
PDFs are the worst offenders. A PDF is a layout format, not a text format. Text extraction from a PDF can produce garbled character orders, broken tables, merged columns, and missing whitespace. Scanned PDFs have no extractable text at all and need OCR before anything else can happen.
Unstructured.io is the standard for production document parsing in 2026. It handles PDFs, DOCX, HTML, images, presentations, and emails. It applies layout detection to separate headers, body text, tables, and figures, and returns clean structured elements rather than raw character streams. For scanned documents, it runs Tesseract OCR automatically.
from unstructured.partition.auto import partition
from unstructured.staging.base import convert_to_dict
# Parse any document type automatically
elements = partition(filename="product_manual.pdf")
# Convert to structured dicts for downstream processing
structured = convert_to_dict(elements)
for item in structured:
if item["type"] in ["NarrativeText", "Title", "ListItem"]:
print(f"[{item['type']}] {item['text'][:100]}")For web content, Firecrawl handles URL-to-Markdown conversion cleanly, stripping navigation, ads, and boilerplate. For large-scale scheduled crawling across thousands of pages, Apify is the production choice.
The output of the parsing step should be clean, normalized Markdown or plain text. Normalize to Markdown before chunking regardless of source format. It makes chunk boundaries predictable and makes it easier to identify structural elements like headers and lists that inform the chunking strategy.
Layer 2: Chunking Boundary
This layer turns parsed documents into retrievable units and attaches source metadata needed later for filtering, citations, and updates. Its architectural contract is to preserve meaning while producing stable identifiers and a reliable link back to the source document.
Chunk size, overlap, semantic boundaries, parent-child retrieval, and document-specific tuning are implementation decisions. Use the practical guide to choosing chunk size for RAG for those choices and their evaluation workflow.
Layer 3: Embedding Contract
This layer converts document chunks and queries into compatible representations for retrieval. Architecturally, it must apply the same model version and preprocessing contract at indexing and query time, expose version information for migrations, and support re-indexing when that contract changes.
Model lists, dimension choices, domain fit, asymmetric retrieval, and benchmarking belong in How Embeddings Work in RAG.
Layer 4: Retrieval Coordination
This optional retrieval layer runs semantic and lexical search paths and presents one merged candidate set to downstream components. Its architecture-level responsibilities are parallel retrieval, consistent document identities across indexes, fusion, filtering, and traceable scores.
Fusion algorithms, candidate depths, implementation code, vendor behavior, and evaluation ablations belong in Hybrid Search for RAG Explained.
Layer 5: Candidate Ordering
This optional layer receives retrieval candidates, reorders them against the original query, and returns a smaller evidence set to context assembly. The surrounding system must preserve candidate IDs, scores, and timing so retrieval failure can be distinguished from ranking failure.
Reranker types, candidate depth, truncation, thresholds, deployment, latency, and evaluation belong in RAG Reranking Explained.
Layer 6: Query Transformation
The user's question is not always a good retrieval query. Short conversational questions, vague references, and underspecified queries all produce poor retrieval results when used directly as the search vector. Query transformation fixes this before retrieval runs.
Query rewriting uses an LLM to expand a vague or ambiguous question into a cleaner, more specific version before embedding it. A user who asks "how does that thing work?" in a customer support context gets their question rewritten to "how does the product's two-factor authentication flow work?" before the vector search runs.
HyDE (Hypothetical Document Embeddings) takes a different approach. Instead of searching for the question, it asks the LLM to generate a hypothetical answer to the question first. That hypothetical answer gets embedded and used as the search query. The distance between a hypothetical answer and real document chunks is smaller in semantic space than the distance between a short question and those same chunks. Retrieval recall improves, particularly for sparse or underspecified queries.
import openai
client = openai.OpenAI(api_key="your-openai-api-key")
def hyde_query(user_question: str, embedding_model: callable) -> list[float]:
"""
HyDE: generate a hypothetical answer, embed it, use that for retrieval.
"""
# Step 1: generate a hypothetical answer
response = client.chat.completions.create(
model="gpt-4o-mini", # fast, cheap model is fine for HyDE
messages=[
{
"role": "system",
"content": (
"Generate a concise, factual answer to the question as if "
"you were writing it for a product documentation page. "
"Do not hedge or qualify. Write the answer directly."
)
},
{"role": "user", "content": user_question}
],
max_tokens=200
)
hypothetical_answer = response.choices[0].message.content
# Step 2: embed the hypothetical answer, not the question
vector = embedding_model(hypothetical_answer)
return vector
# Usage
user_question = "what happens if my payment fails?"
hyde_vector = hyde_query(user_question, your_embed_function)
# Use hyde_vector for retrieval instead of embedding user_question directly
results = vector_db.search(hyde_vector, top_k=20)Sub-question decomposition handles complex multi-hop questions that require information from multiple documents. A question like "how does our refund policy differ between enterprise and starter plan customers?" gets decomposed into "what is the refund policy for enterprise customers?" and "what is the refund policy for starter plan customers?" before retrieval. Both sub-questions run independently and the results get merged before generation.
Layer 7: Agentic RAG
Standard RAG is a single retrieval pass. The question goes in, chunks come back, the model generates an answer. That works well for straightforward factual questions with clear answers in the knowledge base.
Complex questions require multiple retrieval passes. A question that spans multiple documents, requires reasoning across retrieved facts, or cannot be answered by any single chunk does not map cleanly to one retrieve-then-generate loop.
Agentic RAG extends the static pipeline into a dynamic decision-making process. The LLM acts as an orchestrator. It decomposes the question into sub-questions, routes each to an appropriate tool, evaluates whether the retrieved context is sufficient to answer, and performs additional retrieval passes if it is not. It can check its own intermediate results before committing to a final answer.
Agentic RAG Loop
+------------------------------------------+
| User Question |
| | |
| v |
| Planner LLM |
| (decompose into sub-questions) |
| | |
| v |
| Route to retrieval tools: |
| - Vector search (semantic) |
| - Keyword search (BM25) |
| - SQL query (structured data) |
| - Web search (real-time data) |
| | |
| v |
| Retrieve + evaluate sufficiency |
| | |
| |-- Insufficient --> loop back |
| |-- Sufficient --> continue |
| | |
| v |
| Synthesize across retrieved context |
| | |
| v |
| Final Answer with Citations |
+------------------------------------------+Agentic RAG adds model calls, tool calls, and control-flow steps relative to one-pass retrieval, so it normally increases cost and latency. The extra work can be useful for multi-hop or ambiguous questions, but simple lookups may not justify it. Measure task success, latency, and cost against a simpler baseline.
LangGraph and LlamaIndex Workflows are the two standard frameworks for implementing agentic RAG loops in 2026. LangGraph gives you explicit graph-based control over agent state transitions. LlamaIndex Workflows is more ergonomic for retrieval-heavy pipelines.
Layer 8: Evaluation and Observability
The evaluation layer observes retrieval and generation separately so failures can be assigned to the correct component. It consumes representative questions, retrieved evidence, answers, references where available, and trace data; its outputs feed release gates and monitoring.
Metric definitions, dataset construction, RAGAS implementation, thresholds, tracing, and regression workflows belong in RAG Evaluation Engineering.
How the Layers Compound
The layers form a dependency chain: parsing quality affects chunks, chunks and embeddings affect the candidate pool, retrieval and reranking determine the evidence available to generation, and evaluation makes those handoffs observable. A downstream component cannot recover information that an upstream component discarded.
Keep stable IDs and traces across the pipeline so a bad answer can be followed back through generation, ranking, retrieval, indexing, and source parsing. For failure diagnosis, read Why RAG Fails. For storage responsibilities, use Vector Database in RAG; for the architecture decision itself, see RAG vs Fine-Tuning.
Related Reading
Sources
Frequently Asked Questions
What are the main components of a RAG architecture?
A production RAG architecture commonly includes five layers. Document ingestion and chunking converts your raw files into semantically complete text pieces. An embedding model converts each chunk into a numerical vector. A vector database stores and retrieves those vectors. A retrieval layer may combine dense vector similarity with lexical matching and reranking to fetch useful evidence. A generation layer then passes the retrieved chunks as context to the LLM and produces the final answer.
What is HyDE and when should I use it?
HyDE stands for Hypothetical Document Embeddings. Instead of embedding the user's question directly and searching for it, HyDE first asks the LLM to generate a hypothetical answer to the question. That hypothetical answer gets embedded and used as the search query. Because a hypothetical answer is semantically closer to real document chunks than a short question is, retrieval recall improves, particularly for vague or underspecified queries.
What is agentic RAG and when does it justify the extra cost?
Agentic RAG replaces the single retrieve-then-generate step with a dynamic loop where the LLM acts as an orchestrator. It decomposes complex questions into sub-questions, routes each sub-question to the right retrieval tool, evaluates whether the retrieved context is sufficient, and performs additional retrieval passes if needed. Those extra model and tool calls increase token use and latency relative to a one-pass pipeline. They may be justified for multi-hop or ambiguous questions, but the trade-off must be measured for the actual workflow.
Follow on Google
Add as a preferred source in Search & Discover
Add as preferred source
Krunal Kanojiya
Technical Content Writer
Krunal is a technical content writer at Lucent Innovation and a former full-stack developer with professional technology experience since 2021. He publishes source-backed, practical guides on AI engineering, RAG, vector search, data engineering, algorithms, and software development.