Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design
In the first article, we explored why many RAG systems fail in production and established a key principle: retrieval quality determines answer quality. We also introduced the architecture behind production-grade RAG systems and explained why a simple "embeddings + vector database + LLM" pipeline is rarely enough.
In this second part, we'll move one step earlier in the pipeline—to the moment a document first enters your system. We'll examine how documents should be ingested, cleaned, parsed, chunked, and enriched with metadata before a single embedding is generated. These decisions form the foundation of every production RAG system and often have a greater impact on retrieval quality than the choice of embedding model or LLM.
Production RAG Architecture Series
✅ Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search
Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design (you are here)
Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking
Scaling RAG Systems: Production Architecture and Performance Optimization
Evaluating Production RAG Systems: Metrics, Monitoring, and Common Failure Patterns
Chapter 4 — Ingestion: Where Retrieval Really Begins
In production, retrieval does not begin when the user asks a question. It begins the moment a document enters your system. If ingestion is weak, every later step inherits the damage, and no amount of clever prompting will fully recover it.
This is why ingestion is not a “PDF parsing task.” It is the process of turning messy human documents into structured, searchable, trustworthy knowledge.
Why ingestion matters
A document is rarely just plain text. A contract has sections, tables, headers, footers, page numbers, and sometimes scanned signatures. A handbook may contain headings, code blocks, bullet lists, and embedded screenshots. A policy document may look readable to a person but be almost useless to a machine if the structure is flattened too early.
If you simply extract all text and dump it into a chunker, you create noise. The model then sees fragments of meaning instead of complete ideas. That is how a document with a clear rule like “active invoices must be closed before upgrading” can turn into a retrieval result that feels relevant but answers the wrong thing.
What good ingestion does
A good ingestion pipeline does four things:
It preserves structure.
It removes noise.
It extracts metadata.
It prepares the document for retrieval.
That sounds obvious, but most weak systems do only one of these, and often badly. They extract text, maybe split it into chunks, and then assume the rest will work itself out. In practice, ingestion is where the system either becomes searchable knowledge or remains a pile of text.
A simple mental model
Think of ingestion like preparing ingredients before cooking.
You do not throw whole vegetables, packaging, and labels into the pan and hope the meal will sort itself out. You wash, peel, cut, sort, and label things first. In the same way, a document should be cleaned, structured, and annotated before it is indexed.
Here is the basic shape of the pipeline:
Upload
↓
Object Storage
↓
Parser
↓
Cleaner
↓
Structure-aware Chunker
↓
Metadata Extractor
↓
Embeddings
↓
Indexes
Each step exists because raw documents are messy and retrieval systems are picky.
Parsing is not extraction
Parsing is about understanding format, not just getting text out. A PDF parser should know whether the document is scanned or digital, whether the content is laid out in columns, whether tables are visually important, and whether the file contains repeated headers or footers. An HTML parser should preserve semantic tags. A Markdown parser should keep headings, lists, and code blocks intact.
If the parser is dumb, everything after it gets worse.
Cleaning is not optional
After parsing, the document usually still contains junk. You may see:
page numbers,
repeated company names,
“confidential” banners,
broken line wraps,
duplicated paragraphs,
OCR errors.
If you do not remove these, they will be embedded, stored, and later retrieved as if they mattered. That is one of the quietest ways to ruin search quality.
A clean ingestion pipeline removes noise without deleting meaning. That balance matters. Over-cleaning can damage content, while under-cleaning pollutes retrieval.
Structure preservation
Structure is what makes a document usable. Headings tell you what a section is about. Tables encode relationships. Lists show hierarchy. Code blocks preserve exact syntax. Losing that structure is often more harmful than losing a few words.
For example, these two representations are not equivalent:
Payment
Terms Invoices must be paid within 30 days.
Failure to pay may result in service suspension.
and:
Invoices must be paid within 30 days.
Failure to pay may result in service suspension.
The second one is technically readable, but the first one is much more useful because the heading gives the chunk a clear semantic frame. That frame helps retrieval and helps the LLM explain the answer later.
Metadata begins here
Metadata is not something you attach at the end. It should be born during ingestion.
Useful metadata often includes:
document type,
title,
section,
page number,
language,
version,
source system,
department,
creation time,
last updated time.
This metadata is what lets the system answer questions like:
“Show me the latest policy.”
“Search only English documents.”
“Find finance documents.”
“Ignore deprecated versions.”
“Return the clause from page 12.”
Without metadata, everything becomes a vague semantic soup.
A practical example
Suppose you ingest this text:
Title: Pricing Policy
Section: Upgrading Plans
Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.
Section: Downgrading Plans
Downgrading is allowed only if no active trials exist.
A weak pipeline might store it as one long blob. A better pipeline creates structured records like this:
{
"text": "Customers can upgrade from Professional to Enterprise. Important: Active invoices must be closed before upgrading.",
"metadata": {
"title": "Pricing Policy",
"section": "Upgrading Plans",
"document_type": "policy",
"language": "en",
"version": "2.1"
}
}
Now the retriever can filter, rank, and cite this chunk much more intelligently.
OCR and scanned documents
Scanned documents are where ingestion often breaks in production. If text exists only as pixels, you need OCR before anything else can happen. And OCR is not just a fallback feature for edge cases. In many real organizations, a large share of useful knowledge lives in scanned forms, signed contracts, invoices, and old PDFs.
If OCR is poor, the system may miss critical words or mix table cells together. If layout is ignored, the model may see nonsense like one long broken sentence or a table flattened into unreadable text. That is why good ingestion treats OCR and layout extraction as first-class concerns.
The main failure pattern
The most common failure pattern is this:
Document enters the system.
Text is extracted too early.
Structure is lost.
Chunking splits the wrong boundaries.
Metadata is missing or shallow.
Retrieval returns “relevant” but incomplete context.
The LLM answers confidently with the wrong interpretation.
That failure often looks like an AI problem, but it is really an ingestion problem.
What to aim for
Your ingestion pipeline should produce chunks that are:
readable on their own,
tied to a known section or source,
enriched with metadata,
clean enough to embed,
and traceable back to the original document.
If a chunk cannot tell you where it came from and why it exists, it is probably not ready for production.
The core idea
Ingestion is not a support task for retrieval. It is the foundation of retrieval.
That is why production RAG systems spend so much effort on parsing, cleaning, structure, and metadata before they ever touch embeddings. Once that foundation is solid, everything downstream becomes easier, cheaper, and more trustworthy.
Chapter 5 — Chunking: The Heart of Retrieval Quality
Chunking is where most RAG systems quietly fall apart. You can have a solid model, a fast vector database, and a polished prompt, but if the chunks are badly shaped, retrieval will still miss the point.
The reason is simple: retrieval does not search for “truth.” It searches for pieces of text that look useful. If those pieces are incomplete, noisy, or cut in the wrong place, the whole system starts answering with confidence and precision in all the wrong directions.
Why chunking matters
Think about reading a manual where every page was cut into random strips and shuffled. You could still recognize some words, but you would not trust the result. That is exactly what happens when documents are split by token count only.
A good chunk is not just “small enough.” It should be:
semantically coherent,
self-contained enough to understand,
aligned with document structure,
and useful on its own when retrieved.
If a chunk needs three other chunks just to make sense, retrieval quality drops fast.
The wrong way to chunk
The most common beginner approach is fixed-size splitting.
1000 tokens
↓
cut anywhere
↓
chunk
It looks clean in code and terrible in practice. A sentence may get broken in half. A table row may be separated from its header. A section title may end up in one chunk while the content appears in another. Then the retriever sees fragments instead of ideas.
For example:
Original:
"Active invoices must be closed before upgrading from Professional to Enterprise."
Bad chunk 1:
"Active invoices must be closed before upgrading from Professional"
Bad chunk 2:
"to Enterprise."
Neither chunk is ideal. One is incomplete, the other is meaningless. The model may still answer, but it will be guessing.
Fixed-size chunking
Fixed-size chunking is easy to implement and easy to explain, which is why it appears in so many tutorials. It is fine for quick experiments, but it should rarely be your final choice.
Good for:
raw plain text,
prototypes,
quick baselines.
Bad for:
contracts,
docs with headings,
tables,
code-heavy documents,
anything where structure matters.
The problem is not just correctness. Fixed-size splitting also creates retrieval noise. Because chunks are arbitrarily cut, the vector representation becomes less focused. That means the search engine has to work harder just to recover meaning that was destroyed during preprocessing.
Sliding window chunking
Sliding window chunking is the first small improvement. Instead of cutting once and moving on, you keep overlap between adjacent chunks.
Chunk 1: [0..500]
Chunk 2: [400..900]
Chunk 3: [800..1300]
This helps preserve context at the boundaries. If an important sentence gets cut in one chunk, it may still appear in the next one. That alone can significantly improve retrieval.
But overlap is not magic. If the source structure is bad, sliding windows just give you two slightly redundant bad chunks instead of one. It reduces damage; it does not solve the design problem.
Structure-aware chunking
A better approach is to chunk according to the document itself.
If the source is Markdown, use headings. If it is HTML, use semantic tags. If it is a contract, use clauses and sections. If it is a handbook, use chapters and paragraphs.
That gives you chunks like this:
Section: Upgrading Plans
Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.
This is much stronger than a blind token slice because the chunk carries its own context. When the retriever finds it, the LLM sees something that already resembles an answer-ready unit.
Semantic chunking
Semantic chunking goes one step further. Instead of splitting based only on formatting, it tries to split by meaning. The system looks for topic shifts, idea boundaries, or sentence groups that belong together.
This is especially useful when documents are less formal:
product documentation,
internal notes,
policy text,
long articles,
mixed-format knowledge bases.
A semantic chunk should feel like a paragraph a human would naturally quote. If you would not send it to a colleague as a useful excerpt, it is probably not a good chunk.
Parent-child chunking
One of the most effective production patterns is parent-child chunking.
The idea is simple:
search with small child chunks,
answer with larger parent chunks.
Why this works:
small chunks are easier to match precisely,
large chunks provide enough surrounding context for generation.
Example:
Parent chunk:
Section: Upgrading Plans
Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.
If invoices are open, contact billing.
Child chunks:
1. Customers can upgrade from Professional to Enterprise.
2. Important: Active invoices must be closed before upgrading.
3. If invoices are open, contact billing.
The child chunks help retrieval find the relevant idea. The parent chunk gives the model the full answer. This reduces both hallucination and context loss.
Hierarchical chunking
Hierarchical chunking is what happens when you stop thinking in one layer.
Instead of one flat list of chunks, you keep multiple levels:
document,
section,
paragraph,
sentence.
That gives you more control. A broad query can retrieve a section. A precise query can retrieve a paragraph. A citation can point to the exact sentence.
This is especially useful in large corpora where one document may contain many unrelated topics. If you only use flat chunks, the system may know the right document but still miss the right passage.
Recursive chunking
Recursive chunking is a practical compromise. You split by large boundaries first, then smaller ones if needed.
For example:
split by headings,
then paragraphs,
then sentences,
then tokens if absolutely necessary.
This keeps structure intact whenever possible and only falls back to smaller splits when the document is too large. It is a solid default for many production systems because it behaves more like a human editor than a token cutter.
Contextual chunking
Sometimes a chunk is too short to stand alone, even if it is semantically clean. In that case, you add context around it.
Example:
Document title: Pricing Policy
Section: Upgrading Plans
Chunk text:
Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.
This extra context makes the chunk much easier to understand at retrieval time. It also helps the LLM produce better grounded answers because the chunk now carries its own label.
When chunking goes wrong
Chunking failures usually show up in very human ways:
the answer is almost right but misses one rule,
the LLM cites the wrong section,
two different policies get blended into one answer,
a table becomes unreadable,
a “yes” answer appears when the real answer is “yes, but only if...”.
That is why chunking is not a preprocessing detail. It is an information design problem.
Practical guidance
If you are building a real production system, start like this:
preserve structure before splitting,
keep headings with their content,
avoid arbitrary cuts inside tables or code blocks,
use overlap only when needed,
test chunks with real queries, not just with sample text.
A good chunk should answer one natural question or support one natural fact. If it feels like a random slice of text, it probably is.
The main idea
Chunking is not about making text smaller. It is about making knowledge retrievable.
That sounds simple, but it is the difference between a chatbot that sounds smart and a system people actually trust.
Chapter 6 — Metadata: The Difference Between “Similar” and “Correct”
Metadata is where production RAG stops being a clever text search system and starts becoming a trustworthy product. Without metadata, the system can only say, “this looks related”; with metadata, it can say, “this is the right document, from the right version, for the right user, in the right context.”
Why metadata matters
Imagine a supermarket search.
A customer asks for “red dry wine under 20 euros.” If you only use semantic similarity, you may get wines that are red, sweet, expensive, or merely described in a similar style. If you also apply structured filters, you immediately rule out the wrong candidates before retrieval gets expensive or confused. Metadata turns vague similarity into precise constraint handling.
RAG behaves the same way. A user often does not want any semantically related chunk. They want the latest policy, the right language, the correct region, the right team, or the current version of a document. Metadata lets the system narrow the world before it starts ranking chunks.
What metadata actually is
Metadata is not extra decoration. It is structured information attached to documents and chunks so the system can reason about them later.
Useful metadata usually includes:
document title,
document type,
section or heading,
page number,
language,
version,
source system,
department,
created and updated timestamps,
access level,
tags or topic labels.
At the document level, metadata helps you know what something is. At the chunk level, it helps you know where a passage came from and whether it should be considered for a particular query.
Why chunks need metadata too
A chunk without metadata is just floating text. It may be semantically meaningful, but it loses the context that makes retrieval reliable.
For example, this chunk:
Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.
is much more useful if it also carries:
{
"title": "Pricing Policy",
"section": "Upgrading Plans",
"document_type": "policy",
"language": "en",
"version": "2.1"
}
Now the retriever can do more than similarity search. It can filter out old versions, ignore unrelated departments, and keep answers tied to the source section that actually matters.
Metadata is a retrieval control system
Think of retrieval as two decisions:
Which documents are even eligible?
Among those, which chunks are best?
Metadata answers the first question.
A user asking about “the latest finance policy for EMEA” does not need every chunk in your system. They need:
finance documents,
current version,
relevant region,
probably English,
probably policy, not support chat or engineering notes.
If you do this with pure vector similarity, the system has to discover all of that indirectly. That is slow, noisy, and unreliable.
Hard filters and soft filters
Not all metadata is equal.
Some metadata is hard:
tenant ID,
permissions,
access level,
document status,
version validity.
These are non-negotiable. If a document is superseded, deleted, or private, it should not be considered at all.
Other metadata is soft:
topic tags,
department,
region,
audience level,
freshness preference.
These improve relevance, but they should not cause the system to return nothing if they are slightly off.
A good production system treats hard filters as gates and soft filters as ranking hints.
Example: same question, different metadata
Suppose a user asks:
Can a customer upgrade from Professional to Enterprise with active invoices?
The system should not just search for similar wording. It should also know whether the answer belongs to:
a current pricing policy,
a deprecated policy,
a support article,
a legal document,
or an internal note.
Without metadata, all of these may compete in the ranking pool. With metadata, the search space becomes much smaller and much safer.
Query-time filtering
Metadata is most powerful when it is used at query time.
A good flow looks like this:
User Query
↓
Extract likely filters
↓
Apply hard constraints
↓
Search within allowed subset
↓
Rerank results
↓
Send best chunks to LLM
For example, if the query implies:
language = endocument_type = policyversion = latest
then those filters should shape the retrieval before the model sees anything.
That is one of the biggest reasons production RAG feels much better than tutorial RAG: it is not searching the whole world anymore. It is searching the right slice of the world.
Metadata-aware chunking
Metadata is also a chunking problem.
If a paragraph belongs to a section, the chunk should know that. If a table belongs to a page, the chunk should know that too. If a chunk comes from a source that is deprecated, it should carry that status.
Good chunk metadata often includes:
parent document ID,
section name,
page number,
hierarchy path,
extraction method,
language,
freshness state.
This helps in three ways:
filtering,
ranking,
citations.
Without this, the system may retrieve the right words but fail to explain where they came from.
A simple example schema
Here is a practical chunk representation:
{
"id": "pricing_policy_p12_s3_c2",
"text": "Active invoices must be closed before upgrading from Professional to Enterprise.",
"metadata": {
"title": "Pricing Policy",
"section": "Upgrading Plans",
"page": 12,
"document_type": "policy",
"language": "en",
"version": "2.1",
"status": "current",
"department": "billing"
}
}
This chunk is now useful for:
semantic search,
exact filtering,
citations,
debugging,
auditing.
What happens without metadata
Without metadata, retrieval tends to become “close enough” search. That sounds fine until the stakes rise.
Examples of failure:
a user gets an answer from an outdated policy,
the system returns the wrong region,
support sees a marketing document instead of a billing rule,
a legal question is answered from an internal note,
a multilingual corpus returns the wrong language.
The model may still generate a fluent answer, but fluency is not correctness.
The core idea
Metadata is not a nice-to-have. It is the thing that turns retrieval from approximate text similarity into controlled information access.
If chunking defines what a unit of knowledge looks like, metadata defines when and why that unit should be used.
That is why serious RAG systems treat metadata design as a first-class architecture decision, not a last-minute indexing detail.
Continue the Series
This article covered the foundation of every production RAG system: document ingestion, parsing, chunking, and metadata design. These preprocessing decisions determine how effectively your retrieval pipeline can find relevant information long before an LLM generates an answer.
In the next article, we'll move from document preparation to retrieval itself. We'll explore why vector search alone is not enough and how modern production systems combine multiple retrieval techniques to achieve significantly higher accuracy.
Next up:
Part 3 — Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking
We'll cover:
Embeddings and semantic representation
Hybrid search (vector search + BM25)
Query rewriting and optimization
Reranking with cross-encoders
Context compression strategies
Building a production retrieval pipeline
By the end of this series, you'll have a complete engineering framework for designing, building, scaling, and evaluating production-grade Retrieval-Augmented Generation systems.
