Scaling RAG Systems: Production Architecture, Performance, and Cost Optimization
The first three parts of this series covered why production RAG systems fail and how the quality of the data foundation directly affects everything that comes after it. We looked at document ingestion, parsing, chunking, and metadata design—the layers responsible for turning raw information into something a retrieval system can actually work with.
Then we moved into retrieval itself. We saw why vector search alone is often insufficient, how semantic and lexical search complement each other, and how reranking can turn a large set of possible matches into a small set of highly relevant documents.
But even perfect retrieval means nothing if the LLM cannot use it properly.
That is where this part begins.
In Part 4, we move from retrieval to generation. We will look at what happens after the system has found the right chunks: how to compress context without losing meaning, how to construct prompts that keep the model grounded, and how to evaluate whether the entire pipeline is actually working.
We will cover:
Context Compression – how to reduce noise and token cost without losing the evidence that matters.
Prompt Construction – how to build prompts that ground the model, handle missing information, and produce consistent output.
Evaluation – how to measure faithfulness, relevancy, precision, recall, latency, and cost in production.
These are not optional optimizations. They are the layers that turn retrieval into answers people can trust.
Production RAG Architecture Series
✅ Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search
✅ Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design
✅ Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking (you are here)
Scaling RAG Systems: Production Architecture, Performance, and Cost Optimization (you are here)
Evaluating Production RAG Systems: Metrics, Monitoring, and Common Failure Patterns
By the end of this part, you will understand how to turn retrieved context into reliable answers and how to measure whether your system is actually improving over time.
Chapter 10 — Context Compression
Retrieval can give you the right pieces.
That does not mean the LLM should see all of them.
There is a difference between “this chunk is relevant” and “this chunk belongs in the prompt.” A chunk can be relevant and still be too long, too noisy, or too full of unrelated sentences. If you feed everything the retriever found into the model, you pay more, wait longer, and often get worse answers.
That is why context compression exists.
The real problem
A typical RAG pipeline might retrieve 10–20 chunks. Each chunk might be 100–300 words. That is easily thousands of tokens.
But the answer often depends on just a few sentences.
The rest is:
background,
related but not needed,
repeated information,
or noise that survived retrieval.
If the model sees all of that, it has to do extra work. It has to figure out what matters inside the context you gave it. That is exactly the job your retrieval system should have already done.
This is not just about cost. It is about signal-to-noise ratio.
Why compression matters
Compression improves three things:
Cost – fewer tokens means cheaper generation.
Latency – smaller prompts mean faster answers.
Quality – less noise means fewer hallucinations.
That last point is the most important. When the context is cleaner, the model has less room to invent. It has less contradictory information to reconcile. It has fewer chances to latch onto the wrong sentence.
A concrete example
Imagine these retrieved chunks:
Chunk 1:
"Customers can upgrade from Professional to Enterprise. Active invoices must be closed before upgrading. Contact billing if invoices remain open. For more information, see the billing policy."
Chunk 2:
"Billing support is available Monday to Friday. For payment issues, contact billing@example.com. Note that invoice processing may take up to 48 hours."
Chunk 3:
"Downgrading is allowed only if no active trials exist. See the cancellation policy for details. Customers on annual plans have different terms."
Query:
“Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?”
Only one sentence really matters:
“Active invoices must be closed before upgrading.”
The rest is context, but not evidence. Compression should keep that sentence and drop the rest.
Without compression, the model sees all three chunks. It has to figure out which part is relevant. That is extra cognitive load. That is where hallucinations start.
What compression actually does
Context compression is not about making text smaller for the sake of it. It is about keeping the evidence and dropping the noise.
There are three main strategies:
Filtering – remove entire chunks that are not useful.
Extraction – keep only the most relevant sentences from each chunk.
Summarization – compress the remaining text into a shorter form.
Most production systems use filtering first, then extraction, and only use summarization when token budgets are extremely tight.
Filtering
Filtering is the cheapest form of compression. You score each chunk against the query and drop anything below a threshold.
This is usually done with a cross-encoder or a lightweight reranker. The idea is simple: if the chunk is not relevant enough, do not send it to the LLM at all.
from sentence_transformers import CrossEncoder
compressor = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def filter_chunks(query, chunks, threshold=0.5):
pairs = [[query, chunk["text"]] for chunk in chunks]
scores = compressor.predict(pairs)
filtered = [(chunk, score) for chunk, score in zip(chunks, scores) if score > threshold]
filtered.sort(key=lambda x: x[1], reverse=True)
return [chunk for chunk, score in filtered]
This is the first line of defense. It removes entire chunks that are not useful.
Extraction
Extraction is more precise. Instead of dropping entire chunks, you keep only the most relevant sentences inside each chunk.
This is useful when a chunk contains both relevant and irrelevant information. You do not want to lose the relevant part, but you also do not want to send the noise.
A simple approach is to split the chunk into sentences, score each sentence, and keep the top ones.
def extract_sentences(query, chunk, top_n=3):
sentences = chunk["text"].split(". ")
pairs = [[query, sentence] for sentence in sentences]
scores = compressor.predict(pairs)
ranked = sorted(zip(sentences, scores), key=lambda x: x[1], reverse=True)
top_sentences = [sentence for sentence, score in ranked[:top_n]]
return ". ".join(top_sentences)
This keeps the evidence and drops the rest of the chunk.
Summarization
Summarization is the most expensive option. You ask an LLM to rewrite the context into a shorter form.
This is useful when:
you have many chunks,
the token budget is tight,
or the context is repetitive.
But it comes with a cost. Summarization can lose details. It can introduce errors. It can change the meaning.
That is why most production systems use summarization only when necessary.
def summarize_context(query, chunks, llm):
context = "\n\n".join([chunk["text"] for chunk in chunks])
prompt = f"""
Summarize the following context in relation to this query: "{query}"
Context:
{context}
Keep only the information that is directly relevant to answering the query.
Remove any redundant or unrelated information.
Summary:
"""
return llm.generate(prompt)
This is powerful but expensive. Use it carefully.
When to use each strategy
Filtering should always be used. It is cheap and effective. If a chunk is not relevant, do not send it.
Extraction should be used when chunks are long or contain mixed content. It keeps the relevant parts without losing structure.
Summarization should be used when token budgets are tight or when you have many similar chunks. It is expensive but can save a lot of tokens.
The hidden problem: over-compression
Compression can go too far.
If you compress too aggressively, you can:
lose important details,
remove context that the model needs,
or break the structure of the information.
For example, if you extract only one sentence from a chunk that contains a policy with multiple conditions, the model may miss the full picture.
That is why compression should be tuned, not maximized.
Monitoring compression quality
You should track how compression affects your metrics.
Does faithfulness improve?
Does answer relevancy improve?
Does latency decrease?
Does cost decrease?
If compression improves cost and latency but hurts quality, you are compressing too much.
Example: full compression pipeline
def compress_context(query, chunks, llm=None, token_budget=2000):
# Step 1: Filter chunks
filtered = filter_chunks(query, chunks, threshold=0.4)
# Step 2: Extract sentences from each chunk
extracted = []
for chunk in filtered:
extracted_text = extract_sentences(query, chunk, top_n=3)
extracted.append({"text": extracted_text})
# Step 3: Check token budget
total_tokens = sum(len(chunk["text"].split()) * 1.3 for chunk in extracted)
if total_tokens > token_budget and llm:
# Step 4: Summarize if over budget
context = summarize_context(query, extracted, llm)
return [context]
return extracted
This is a simple pipeline that combines all three strategies.
When not to compress
Compression is not always the right move.
You should be careful when:
chunks contain code,
chunks contain tables,
chunks are already short,
or the context is already tight.
In those cases, aggressive compression can remove important structure or details.
The real lesson
Context compression is not an optional optimization. It is a way to make sure the LLM sees the right evidence, not just a lot of text.
If retrieval is the net, compression is the hand that removes the fish you do not need.
The goal is not to make the context as small as possible. The goal is to make it as useful as possible.
Chapter 11 — Prompt Construction
A good prompt cannot save bad retrieval.
A bad prompt can ruin good retrieval.
That is the entire chapter.
The prompt is where everything comes together: the query, the retrieved context, the instructions, the output format, and the guardrails. If any of those pieces is weak, the answer will be weak. But if retrieval is already broken, even the best prompt will just make the wrong answer sound more confident.
The role of the prompt
The prompt has three jobs:
Ground the model – make it clear that the answer must come from the provided context.
Structure the answer – define how the output should look.
Set guardrails – tell the model what to do when the context is insufficient.
That sounds simple, but most production prompts fail on at least one of these.
Why grounding matters
The most common failure mode is when the model ignores the context and answers from its own knowledge. That is why the instruction “answer ONLY using the provided context” is not decorative. It is the core of RAG faithfulness.
Without that instruction, the model may:
invent details,
mix policies from different documents,
or confidently state something that is not in the context.
That is why grounding is the first priority.
A basic RAG prompt structure
You are a helpful assistant that answers questions based ONLY on the provided context.
Context:
{retrieved_chunks}
Question:
{query}
Instructions:
- Answer using only the information in the context.
- If the answer cannot be found, say "I don't have enough information."
- Cite the source document and section when possible.
- Keep the answer concise and direct.
Answer:
This is the minimal viable shape. It tells the model what to do, what not to do, and how to handle uncertainty.
The anatomy of a good prompt
A production prompt usually has these components:
System role – defines the assistant's behavior.
Context – the retrieved chunks.
Question – the user query.
Instructions – how to answer.
Output format – how the answer should look.
Guardrails – what to do when context is insufficient.
Each component matters.
System role
The system role sets the tone. It tells the model what kind of assistant it is.
You are a helpful assistant that answers questions based ONLY on the provided context.
You do not use outside knowledge.
You do not invent information.
If the answer is not in the context, you say so.
This is not just flavor text. It is a constraint that shapes the entire generation.
Context
The context is the retrieved chunks, usually after compression.
How you format the context matters.
A common pattern is to number each chunk and include metadata:
Context:
[1] Document: Pricing Policy, Section: Upgrading Plans
Customers can upgrade from Professional to Enterprise. Active invoices must be closed before upgrading.
[2] Document: Billing Policy, Section: Payment Terms
Invoices must be paid within 30 days. Failure to pay may result in service suspension.
[3] Document: Support Policy, Section: Contact
Billing support is available Monday to Friday. Contact billing@example.com.
This format makes it easier for the model to cite sources and for you to debug later.
Question
The question should be clear and isolated from the context.
Question:
Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?
Do not mix the question with the context. Keep them separate.
Instructions
Instructions tell the model how to answer.
Good instructions:
are specific,
are actionable,
and cover edge cases.
Instructions:
- Answer using only the information in the context.
- If the answer cannot be found, say "I don't have enough information to answer this question from the provided documents."
- Cite the source document and section when possible.
- Keep the answer concise and direct.
- Do not repeat the context verbatim.
Notice the explicit instruction for missing information. That is critical.
Output format
The output format depends on your use case.
For a chatbot:
Answer in 2-3 sentences.
For a structured API:
Answer in JSON format:
{
"answer": "...",
"citations": [
{"document": "...", "section": "..."}
]
}
For a technical assistant:
Answer in bullet points.
Include code examples when relevant.
The format should match the product, not the model.
Guardrails
Guardrails are the safety net.
They tell the model what to do when:
the context is insufficient,
the question is ambiguous,
or the answer requires external knowledge.
Guardrails:
- If the context does not contain the answer, say "I don't have enough information."
- If the question is ambiguous, ask for clarification.
- Do not provide legal, medical, or financial advice.
- Do not speculate.
These are not optional. They are the difference between a system that knows its limits and one that hallucinates confidently.
Example code
def build_prompt(query, chunks):
# Format context with metadata
context_parts = []
for i, chunk in enumerate(chunks, 1):
metadata = chunk.get("metadata", {})
doc = metadata.get("title", "Unknown")
section = metadata.get("section", "Unknown")
context_parts.append(f"[{i}] Document: {doc}, Section: {section}\n{chunk['text']}")
context = "\n\n".join(context_parts)
prompt = f"""
You are a helpful assistant that answers questions based ONLY on the provided context.
You do not use outside knowledge.
You do not invent information.
If the answer is not in the context, you say so.
Context:
{context}
Question:
{query}
Instructions:
- Answer using only the information in the context.
- If the answer cannot be found, say "I don't have enough information to answer this question from the provided documents."
- Cite the source document and section when possible.
- Keep the answer concise and direct.
- Do not repeat the context verbatim.
Output format:
- Answer in 2-3 sentences.
- Include citations with document title and section.
Answer:
"""
return prompt
This is the basic shape. Production prompts often add more guardrails, but the core idea is the same.
Common prompt mistakes
Mistake 1: No grounding instruction
Bad:
Context: {context}
Question: {query}
Answer:
The model has no instruction to stay in the context. It will answer from its own knowledge.
Mistake 2: No handling for missing information
Bad:
- Answer using only the information in the context.
What if the context does not have the answer? The model will guess.
Mistake 3: No output format
Bad:
- Answer the question.
The model does not know how long the answer should be or what format to use.
Mistake 4: Too much context
If you send 10 chunks without compression, the model has to find the signal in the noise. That is when hallucinations start.
Testing prompts
Prompts should be tested like code.
Build a small dataset of queries and expected answers. Run your prompt against them. Check:
Does the model stay grounded?
Does it handle missing information?
Does it follow the output format?
Does it cite sources correctly?
This is not optional. It is how you catch prompt regressions.
The core lesson
The prompt is not where you fix retrieval. It is where you make sure the retrieval you have is used correctly.
If the context is good, a good prompt makes the answer better.
If the context is bad, a good prompt just makes the wrong answer clearer.
A prompt is not magic. It is a set of instructions. And like any instructions, it only works if the foundation is solid.
Chapter 12 — Evaluation
You cannot improve what you do not measure.
That is the entire chapter.
Most RAG systems fail in production not because the components are broken, but because nobody knows when they are getting worse. Evaluation is the layer that turns subjective “feels better” into objective “is better.”
Without evaluation, you are flying blind. You change the chunking strategy. You switch the embedding model. You tweak the prompt. And then what? You look at a few queries and say “seems better”? That is not engineering. That is guessing.
Why evaluation matters
Evaluation gives you:
baselines,
regression detection,
quality thresholds,
and a way to compare changes.
It turns RAG from a black box into a system you can actually improve.
The core metrics
Most production systems track a small set of metrics:
Faithfulness – is the answer grounded in the context?
Answer Relevancy – does the answer address the question?
Context Precision – how much of the retrieved context is relevant?
Context Recall – did retrieval find the right content?
Latency – how long does each step take?
Cost – how many tokens per query?
These metrics cover both retrieval and generation.
Faithfulness
Faithfulness measures whether the claims in the answer are supported by the context.
If the model says something that is not in the context, faithfulness drops. That is the metric that catches hallucinations.
For example:
Context:
"Active invoices must be closed before upgrading."
Answer:
"Yes, customers can upgrade while keeping active invoices."
Faithfulness: Low (the answer contradicts the context)
A high faithfulness score means the model is staying grounded. A low score means it is inventing or contradicting.
Answer Relevancy
Answer relevancy measures whether the answer actually addresses the question.
A model can be faithful but irrelevant. For example, it can faithfully repeat the context without answering the query. Relevancy catches that.
Question:
"Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?"
Answer:
"Customers can upgrade from Professional to Enterprise. Active invoices must be closed before upgrading."
Relevancy: High (the answer addresses the question)
Answer:
"Active invoices must be closed before upgrading. For more information, contact billing."
Relevancy: Medium (partial answer, no direct yes/no)
Answer:
"Billing support is available Monday to Friday."
Relevancy: Low (does not address the question)
Context Precision
Context precision measures how much of the retrieved context is relevant.
If you retrieve 10 chunks but only 1 is relevant, precision is low. That means you are wasting tokens and confusing the model.
Retrieved: 10 chunks
Relevant: 2 chunks
Precision: 0.2
High precision means your retrieval is focused. Low precision means you are sending too much noise.
Context Recall
Context recall measures whether the right content was retrieved at all.
If the answer exists in your corpus but retrieval did not find it, recall is low. That is a retrieval problem, not a generation problem.
Total relevant chunks in corpus: 5
Retrieved relevant chunks: 3
Recall: 0.6
High recall means your retrieval is finding the right content. Low recall means you are missing it.
Latency and Cost
Latency and cost are operational metrics, but they matter.
Latency – how long does each step take?
Retrieval time
Reranking time
Compression time
Generation time
Cost – how many tokens per query?
Input tokens
Output tokens
Total cost per query
If your system is accurate but takes 10 seconds per query, users will not use it. If it is accurate but costs $1 per query, you will not scale.
Example code with Ragas
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
# Your dataset should have:
# - question
# - answer
# - contexts (list of retrieved chunks)
# - ground_truth (optional, for some metrics)
results = evaluate(
dataset=your_dataset,
metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)
print(results)
This is the minimal setup. Production systems add more metrics, but these four are the core.
Building a golden dataset
Evaluation requires a dataset of queries with expected answers or relevant context.
Most teams build 100–200 examples that cover:
common queries,
edge cases,
difficult queries,
and known failure modes.
This dataset becomes the baseline for every change.
from datasets import Dataset
data = {
"question": [
"Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?",
"How do I reset my password?",
"What is the refund policy for annual plans?"
],
"answer": [
"No. Active invoices must be closed before upgrading from Professional to Enterprise.",
"Go to Settings → Security → Reset Password. You'll receive an email with a link.",
"Annual plans are non-refundable. You can downgrade at the end of the billing cycle."
],
"contexts": [
["Active invoices must be closed before upgrading."],
["Go to Settings → Security → Reset Password."],
["Annual plans are non-refundable."]
],
"ground_truth": [
"No. Active invoices must be closed before upgrading.",
"Go to Settings → Security → Reset Password.",
"Annual plans are non-refundable."
]
}
dataset = Dataset.from_dict(data)
This is the starting point. You expand it over time.
Production monitoring
Evaluation does not stop at deployment.
Production systems should:
sample real queries,
run them through evaluation,
track metrics over time,
and alert when quality drops.
That is how you catch regressions before users do.
def monitor_production(queries, answers, contexts):
for query, answer, context in zip(queries, answers, contexts):
score = evaluate_single(query, answer, context)
log_metric(score)
if score["faithfulness"] < 0.7:
alert("Low faithfulness detected")
This is the basic idea. You track metrics in production and alert when they drop below thresholds.
Setting thresholds
Thresholds depend on your use case, but here are some rough guidelines:
Faithfulness: 0.8 or higher for production
Answer Relevancy: 0.7 or higher
Context Precision: 0.5 or higher
Context Recall: 0.7 or higher
These are not universal. They are starting points.
The evaluation loop
Evaluation is not a one-time thing. It is a loop:
Build a golden dataset.
Run evaluation.
Identify weak points.
Make changes.
Re-run evaluation.
Deploy if metrics improve.
Monitor in production.
Repeat.
This is how you actually improve a RAG system over time.
Common evaluation mistakes
Mistake 1: No golden dataset
You cannot evaluate without a baseline. If you do not have a dataset of queries and expected answers, you are just guessing.
Mistake 2: Only evaluating on easy queries
If your dataset only contains easy queries, you will not catch edge cases. Include difficult queries, ambiguous queries, and known failure modes.
Mistake 3: Not monitoring in production
Evaluation in development is not enough. You need to monitor real queries in production to catch regressions.
Mistake 4: Ignoring latency and cost
Accuracy is not the only metric. If your system is accurate but too slow or too expensive, it will not scale.
The core lesson
Evaluation is not a nice-to-have. It is the layer that makes RAG engineering possible.
Without it, you are just guessing.
With it, you can actually build something that gets better over time.
Evaluation turns RAG from a black box into a system you can measure, improve, and trust.
Continue the Series
This article focused on scaling production RAG systems: large-scale architecture, ingestion and retrieval pipelines, vector database performance, caching, async processing, and cost optimization. Together, these techniques help RAG systems remain fast, reliable, and cost-efficient as the amount of data and traffic grows.
But a scalable system means nothing if you cannot measure whether it is actually working correctly.
In the next article, we will move from infrastructure to evaluation. We will explore how to measure RAG quality in production, what metrics matter, how to monitor for regressions, and what failure patterns to watch for.
Next up:
Part 5 — Evaluating Production RAG Systems: Metrics, Monitoring, and Common Failure Patterns
We will cover:
RAG evaluation metrics (faithfulness, relevancy, precision, recall)
Building golden datasets
Production monitoring and alerting
Common failure patterns and how to catch them
Setting quality thresholds
Continuous evaluation loops
By the end of this series, you will have a complete engineering framework for designing, building, scaling, and evaluating production-grade Retrieval-Augmented Generation systems.
