How to Choose Chunk Size for RAG: A Practical Guide

If you need a practical starting point, test 400 to 600-token chunks with about 10% overlap for a general text knowledge base. But do not treat that range as the best chunk size for every RAG system. Product documentation, contracts, source code, transcripts, and financial reports have different answer boundaries.

The right chunk size is the smallest chunk that contains enough evidence to answer the query correctly without surrounding it with unrelated information.

That definition matters more than any universal number. NVIDIA's multi-dataset evaluation found that fact-based questions often worked well with 256 to 512-token chunks, while analytical questions benefited from 1,024-token or page-level chunks. Even three financial datasets in the same evaluation produced different winners. Chunk size is a parameter to measure, not a constant to copy.

This guide gives you a starting configuration, explains the trade-offs, and shows how to choose chunk size for RAG with evidence from your own documents and queries. For the wider ingestion and retrieval pipeline, start with RAG architecture explained.

What Does Chunk Size Mean in RAG?

Chunk size is the maximum amount of content placed in one retrievable unit before that unit is converted into an embedding and stored in a vector index.

Imagine a product manual with 80 pages. Embedding the entire manual as one vector would compress installation instructions, error codes, safety warnings, and warranty rules into a single representation. A query about error code E17 would compete with every other topic in the document.

Chunking produces smaller retrieval units:

Document
  -> Section: Installation
  -> Section: Error E17
  -> Section: Warranty

Each section
  -> embedding
  -> vector index
  -> independently retrievable result

When a user asks a question, the retriever returns the chunks that appear most relevant. Those chunks become context for the language model.

Chunk size can be measured in:

• Tokens, which align with embedding and language-model limits
• Characters, which are simple and deterministic but language-dependent
• Words or sentences, which are easier for humans to interpret
• Structural units, such as headings, paragraphs, clauses, pages, functions, or conversation turns

In production, structure should determine the boundaries and tokens should enforce the maximum size. A 600-token limit is useful. Splitting exactly at token 600, halfway through a table or procedure, is not.

The Short Answer: Which Chunk Size Should You Start With?

Use this as an experiment plan, not as a final answer:

These are starting candidates. They are intentionally ranges rather than promises.

The NVIDIA chunking study tested token sizes from 128 to 2,048 with page- and section-level alternatives. Page-level chunks had the best average accuracy across its datasets, but 512 tokens won for one financial dataset and 1,024 won for another. A controlled 2026 study likewise found chunking effects were non-linear for code completion. The evidence does not support one global optimum.

Why Chunk Size Changes Retrieval Quality

Chunk size controls a trade-off between precision and context.

Chunks That Are Too Small

Small chunks can match a narrow query precisely, but they often lose the information around the answer.

Suppose a policy contains this passage:

Enterprise customers can request a refund within 30 days.
This policy applies only when fewer than 20% of licensed seats were activated.
Refund requests must include the original invoice.

If each sentence becomes a separate chunk, a search for the refund period may retrieve the first sentence without the eligibility condition. The answer is technically present but operationally incomplete.

Small chunks can also:

• Create more embeddings and increase indexing cost
• Require a larger retrieval top_k to reconstruct the answer
• Remove headings that explain what a paragraph refers to
• Separate a table from its caption or column definitions
• Produce fragments that are meaningless outside the source document

Chunks That Are Too Large

Large chunks preserve context, but they mix more topics into one embedding. The exact answer can become a small signal inside a broad section.

That produces a different set of problems:

• Lower retrieval precision because unrelated text influences similarity
• More irrelevant tokens passed to the language model
• Fewer distinct sources fitting inside the context budget
• Higher reranking and generation cost
• Greater risk that the model overlooks the answer within a long passage

LangChain's official documentation explains the underlying goal well: split documents into pieces that can be retrieved independently while still fitting within the model's context window. Its recursive splitter preserves paragraphs and sentences where possible, which is a better baseline than cutting raw text at an arbitrary character.

Choose the Retrieval Unit Before Choosing the Number

The strongest way to choose chunk size is to ask:

What is the smallest unit in this corpus that can answer a real user question correctly?

That unit changes by document type.

Product and API Documentation

Use headings and subsections as primary boundaries. Keep a procedure, its prerequisites, and its warnings together. Keep code blocks with the paragraph that explains them.

If a section is too large, split it recursively by paragraph and sentence. Prepend the document title and heading path to every child chunk:

Document: Payments API
Section: Authentication > Token expiration

Access tokens expire after 24 hours...

The heading gives a small child chunk enough context to be understandable and improves exact-topic retrieval.

Policies, Contracts, and Legal Documents

Use clauses or subsections rather than uniform windows. Definitions should remain available to the clauses that use them. When a clause is small but depends on a larger section, use parent-child retrieval: retrieve the precise child, then send its parent section to the language model.

Financial Reports and Complex PDFs

Do not split tables row-by-row without their headers. Keep a complete table when it fits, or repeat the title, column headers, units, and reporting period in every row group.

Research on financial reports found that chunking by structural element can improve retrieval compared with treating all text as equivalent paragraphs. PDF extraction quality is part of this decision: a perfect token size cannot repair a table whose columns were parsed in the wrong order.

Source Code

Split around syntactic units such as declarations, functions, classes, and modules. Include the qualified symbol name, file path, imports, and relevant class context as metadata.

Do not assume function-level chunks always win. A large controlled study of code RAG found function chunking underperformed sliding-window and syntax-aware alternatives on its tested code-completion benchmarks. Code retrieval should be evaluated against the tasks you actually support: explanation, completion, bug search, or cross-file navigation.

Conversations and Support Tickets

Preserve speaker labels and turn order. A user message and the agent's answer usually form a better retrieval unit than isolated sentences. For longer threads, create short turn windows and store ticket-level metadata such as product, issue type, status, and date.

How Much Chunk Overlap Should You Use?

Chunk overlap repeats content from the end of one chunk at the beginning of the next. Its purpose is to protect information that crosses an artificial boundary.

Chunk A: tokens 1-512
Chunk B: tokens 462-974

Overlap: 50 tokens, approximately 10%

A practical starting point is 0% to 10% overlap:

• Use 0% when chunks follow complete semantic or structural units
• Use about 10% for token windows and recursive splitting
• Increase it only when evaluation shows boundary-related misses
• Reduce it when retrieved results contain repeated versions of the same passage

Overlap has real costs. It creates more vectors, increases embedding and storage expense, and can cause near-duplicate chunks to occupy several of the top results. More overlap can therefore lower the diversity of evidence presented to the model.

Evidence on overlap is mixed, which is another reason to test it. NVIDIA reported that 15% performed best in a limited overlap experiment using 1,024-token chunks on FinanceBench. A separate systematic analysis found no measurable benefit from overlap in its question-answering setup. Both results can be valid because the corpora, splitters, retrievers, and questions differ.

If natural boundaries already protect complete ideas, overlap may be solving a problem you no longer have.

Fixed, Recursive, Semantic, or Parent-Child Chunking?

Chunk size cannot be separated from chunking strategy.

For most text collections, recursive structure-aware chunking is the best baseline. LangChain also recommends starting with its recursive splitter for general use because it attempts to keep paragraphs, then sentences, then words intact.

Parent-child retrieval is especially useful when you cannot make one chunk serve two opposing goals:

1. Create small child chunks for precise embedding and retrieval.
2. Store the identifier of each larger parent section.
3. Retrieve using the child chunks.
4. Replace or expand each winning child with its parent before generation.

This lets a 200-token answer-bearing passage rank precisely while the language model receives the full 800-token section needed to interpret it.

A Practical LangChain Baseline

Here is a token-aware recursive baseline for English documentation:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    model_name="text-embedding-3-small",
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " ", ""],
)

documents = splitter.create_documents(
    texts=[markdown_text],
    metadatas=[{
        "source": "payments-api.md",
        "document_title": "Payments API",
        "version": "2026-07",
    }],
)

for index, document in enumerate(documents[:3]):
    print(index, len(document.page_content), document.metadata)

Two cautions:

First, chunk_size means different things depending on the splitter and length function. A character-based setting of 1,000 is not a 1,000-token chunk. The LangChain splitter API defines chunk size through its configured length function, so confirm what your implementation is measuring.

Second, this code is a baseline. It does not preserve complex tables, images, cross-references, or code syntax by itself. Parse those elements deliberately before splitting.

How to Find the Best Chunk Size With an Evaluation

The most reliable answer comes from a controlled evaluation on your own corpus.

Step 1: Build a Representative Question Set

Start with 50 to 100 questions if possible. Include:

• Common fact lookups
• Questions whose wording differs from the source
• Questions requiring a condition or exception
• Multi-step and analytical questions
• Exact identifiers, error codes, dates, and product names
• Questions that should return no answer

For each answerable question, record the source passage or document that contains the evidence. Use real support, search, or product queries when they are available. Synthetic questions can expand coverage, but manually review them.

Step 2: Define Candidate Configurations

For a general knowledge base, test a small grid:

A: 256 tokens, 0 overlap
B: 256 tokens, 25 overlap
C: 512 tokens, 0 overlap
D: 512 tokens, 50 overlap
E: 1024 tokens, 0 overlap
F: section-aware chunks capped at 1024 tokens
G: 256-token children with section-level parents

Keep everything else fixed: parser, embedding model, vector index, distance metric, metadata filters, top_k, reranker, and generation prompt. Otherwise, you will not know whether chunking caused the difference.

Step 3: Measure Retrieval Before Generation

Use retrieval metrics first:

• Hit rate at k: Did any of the top k chunks contain the answer?
• Recall at k: How much of the required evidence appeared in the top k?
• Mean reciprocal rank: How high did the first relevant chunk rank?
• nDCG: Were highly relevant chunks ranked above partially relevant ones?
• Context precision: How much retrieved text was actually relevant?
• Duplicate rate: How many results repeated substantially the same content?

For a question set with one known relevant chunk per question, this small evaluation function provides a starting point:

def evaluate_rankings(results_by_query, relevant_ids_by_query, k=5):
    hits = 0
    reciprocal_ranks = []

    for query_id, ranked_ids in results_by_query.items():
        relevant_ids = set(relevant_ids_by_query[query_id])
        top_ids = ranked_ids[:k]

        if any(item_id in relevant_ids for item_id in top_ids):
            hits += 1

        first_relevant_rank = next(
            (rank for rank, item_id in enumerate(ranked_ids, start=1)
             if item_id in relevant_ids),
            None,
        )
        reciprocal_ranks.append(
            1 / first_relevant_rank if first_relevant_rank else 0
        )

    query_count = len(results_by_query)
    return {
        f"hit_rate@{k}": hits / query_count,
        "mrr": sum(reciprocal_ranks) / query_count,
    }

Step 4: Measure End-to-End Answer Quality

Retrieving the right passage does not guarantee a correct answer. Compare:

• Factual correctness
• Faithfulness to retrieved evidence
• Completeness
• Citation correctness
• Abstention when evidence is missing
• Prompt tokens, latency, and cost

A clinical RAG study illustrates why end-to-end testing matters: its adaptive chunking configuration achieved higher retrieval metrics and higher answer accuracy than the fixed-size baseline. The result belongs to that clinical setup, not every corpus, but it demonstrates that chunking can affect both retrieval and final answers.

Step 5: Inspect Failures Manually

Metrics tell you which configuration won. Failed queries tell you what to change.

For every miss, ask:

• Was the source parsed correctly?
• Did a boundary split the answer from a condition or heading?
• Was the chunk relevant but too broad?
• Did metadata filtering remove the correct document?
• Did the embedding model fail on domain terminology?
• Did duplicate overlapping chunks crowd out other evidence?
• Was the correct chunk retrieved but lost during reranking?
• Did the language model ignore evidence that was present?

Do not keep adjusting chunk size when the real issue is parsing, metadata, embeddings, filtering, or ranking. Read why RAG fails for the wider diagnostic process and RAG evaluation engineering for retrieval and generation metrics.

How Chunk Size Interacts With Top-K and Context Budget

Chunk size is only one part of the retrieval budget.

If you retrieve k chunks of average size s, the raw retrieved context is approximately:

retrieved tokens = k x s

Five 1,000-token chunks consume about 5,000 tokens before the system prompt, conversation history, citations, and answer. Ten 250-token chunks consume about 2,500 tokens but may include more diverse sources.

Changing chunk size without retuning top_k makes comparisons misleading. Smaller chunks may need a higher top_k to assemble complete evidence. Larger chunks may need a lower top_k to stay inside the context budget.

A good evaluation therefore records the entire configuration:

parser + strategy + chunk size + overlap + embedding model
+ retriever + top_k + reranker + generation context budget

If retrieval precision is low, do not automatically make chunks smaller. Consider hybrid search for exact terms and a reranker for better ordering.

Common Chunk-Size Mistakes

Copying a Framework Default

A default proves that the tutorial runs. It does not prove that the configuration matches your documents or questions.

Confusing Characters With Tokens

chunk_size=1000 may mean characters in one splitter and tokens in another. Record the unit in configuration names and experiment reports.

Splitting Before Parsing the Document

Headers, footers, navigation, broken columns, and repeated page text contaminate embeddings. Clean and structure the source before optimizing chunk size.

Breaking Tables, Lists, and Code Blocks

These elements depend on internal structure. Keep headers with table rows, steps with their prerequisites, and code with the explanation or symbol metadata needed to interpret it.

Adding Too Much Overlap

Overlap can protect a boundary, but it can also make the top five results five versions of the same paragraph. Track duplicate rate alongside recall.

Evaluating Only the Final Answer

An answer score does not show whether the failure happened in parsing, retrieval, reranking, or generation. Evaluate retrieval and generation separately.

Using One Configuration for Every Document Type

A mixed corpus may need routing. Use one chunking policy for API documentation, another for tables, and another for support conversations. Store the content type in metadata so retrieval can filter or route correctly.

A Production Checklist

Before you commit to a RAG chunk size, confirm that:

• The chunk is understandable without opening the original document
• Headings and source metadata travel with the chunk
• Answers are not separated from their conditions or exceptions
• Tables, lists, code blocks, and conversation turns remain structurally valid
• Chunk size is measured in a clearly documented unit
• The embedding model accepts the full chunk length
• Overlap solves observed boundary misses rather than following a habit
• top_k and context budget are evaluated with the chunk size
• Retrieval metrics are measured on representative questions
• Failed retrievals are manually inspected
• Index size, latency, and cost are recorded
• Re-indexing is versioned so configurations can be compared or rolled back

Final Recommendation

For a new text-based RAG system, start with structure-aware recursive splitting, cap chunks at 512 tokens, and use about 50 tokens of overlap only where natural boundaries are not reliable. Then compare that baseline against 256-token, 1,024-token, and structure-level alternatives.

Choose the configuration that retrieves complete answer-bearing evidence at the highest useful rank with the least noise. If small chunks retrieve accurately but lack context, use parent-child retrieval rather than making every indexed chunk larger.

The real answer to “how to choose chunk size for RAG” is not 256, 512, or 1,024. It is a controlled decision based on your documents, your questions, your retrieval metrics, and your context budget.

Continue with RAG architecture explained to see where chunking fits alongside embeddings, hybrid retrieval, reranking, and evaluation. If you are still building the foundation, read what RAG is and how embeddings work in RAG.