# Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking The first two parts of this series covered why production RAG systems fail and how the quality of the data foundation directly affects everything that comes after it. We looked at document ingestion, parsing, chunking, and metadata design—the layers responsible for turning raw information into something a retrieval system can actually work with. But even perfectly processed documents are useless if retrieval cannot find the right information. In this third part, we'll move into the retrieval layer itself. We'll look at why vector search alone is often insufficient, how semantic and lexical search complement each other, and how reranking can turn a large set of possible matches into a small set of highly relevant documents. We'll also cover query optimization, metadata filtering, and context compression—key techniques for building retrieval pipelines that perform reliably on real-world queries. ## Production RAG Architecture Series 1. ✅ **Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search** 2. ✅ **Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design** 3. **Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking** *(you are here)* 4. Scaling RAG Systems: Production Architecture, Performance, and Cost Optimization 5. Evaluating Production RAG Systems: Metrics, Monitoring, and Common Failure Patterns * * * ## Chapter 7 — Embeddings Embeddings are not magic. They are coordinates. That is the whole trick. A piece of text goes in, a vector comes out, and now similar meanings sit close to each other in space. If chunking decides what the system sees, embeddings decide how it remembers it. That sounds abstract until you try to build retrieval on top of it. Then it becomes the center of the whole system. * * * ## What an embedding really is Imagine a map. On that map: * “dog” sits near “wolf”. * “invoice” sits near “payment”. * “upgrade” sits near “billing policy”. * “password reset” sits somewhere else. The model is not understanding meaning the way a human does. It is learning a geometry where related things end up near each other. That geometry is what retrieval uses later. And that is why embeddings matter so much. If the geometry is good, retrieval feels smart. If the geometry is bad, everything downstream starts guessing. * * * ## Why this is not enough This is where people usually make the first mistake. They think: > “If I use a good embedding model, retrieval will work.” It won’t. A good embedding model can only work with the text you give it. If the chunk is messy, too broad, too short, or stuffed with unrelated ideas, the vector will still be messy. Just in a more expensive way. A bad chunk becomes a bad vector. A bad vector becomes a bad candidate. A bad candidate becomes a confident wrong answer. * * * ## A concrete example Take these chunks: ```plaintext 1. Active invoices must be closed before upgrading. 2. Customers can upgrade from Professional to Enterprise. 3. How to reset your password. 4. Downgrading is allowed only if no active trials exist. ``` A decent embedding model should understand that 1 and 2 belong near upgrade-related questions, while 3 is clearly off in another part of the world. That sounds obvious, but in real systems it gets messy fast. Because now you have: * legal docs, * support docs, * product policies, * release notes, * tables, * code snippets, * and old versions of the same document all mixed together. At that point embeddings are not a detail anymore. They are the shape of the search space. * * * ## What makes a good embedding model A good model for production should: * understand your language, * behave well on short queries, * not collapse technical terms into generic similarity, * and work on your actual domain, not just “general text.” A model that is decent for blog posts may be weak for: * policy documents, * multilingual corpora, * technical manuals, * product docs with version numbers, * or support data full of exact identifiers. So the real question is not “which embedding model is popular?” The real question is “which model gives me the right geometry for my corpus?” ## Example code python ```python from sentence_transformers import SentenceTransformer import numpy as np model = SentenceTransformer("all-MiniLM-L6-v2") chunks = [ "Active invoices must be closed before upgrading.", "Customers can upgrade from Professional to Enterprise.", "How to reset your password.", "Downgrading is allowed only if no active trials exist." ] vectors = model.encode(chunks, normalize_embeddings=True) def cosine(a, b): return float(np.dot(a, b)) query = "Can Enterprise customers upgrade directly from Professional while keeping active invoices?" query_vector = model.encode([query], normalize_embeddings=True)[0] ranked = [] for chunk, vector in zip(chunks, vectors): score = cosine(query_vector, vector) ranked.append((chunk, score)) ranked.sort(key=lambda x: x[1], reverse=True) for chunk, score in ranked: print(f"{score:.4f} | {chunk}") ``` This is the smallest possible version of the idea. Query becomes a vector. Chunk becomes a vector. Similarity becomes a number. The number is not truth. It is only a signal. But in a good system, that signal is useful enough to move the right chunk to the top. ## Why chunk shape changes embedding quality A short query and a long chunk do not behave the same way. A query like: > “Enterprise upgrade active invoices” is compact and vague. A chunk like: > “Customers can upgrade from Professional to Enterprise. Active invoices must be closed before upgrading. Contact billing if invoices remain open.” contains multiple ideas. The embedding becomes a compressed summary of all of that. If the chunk contains too many unrelated ideas, the vector turns into an average of everything, which is another way of saying it gets blurrier. That is why embeddings and chunking are inseparable. You cannot fix one without thinking about the other. * * * ## The domain problem General embeddings are often good enough to impress in demos. Production is where they start revealing their limits. A support system might need to understand: * plan names, * billing states, * status codes, * product tiers, * policy phrases, * internal jargon. A generic model may know the words, but not the importance of those words in your system. That is why evaluation on real queries matters. Not benchmark queries. Your queries. * * * ## The real lesson Embeddings are not magic meaning detectors. They are a way to build a space where retrieval can do its job. If the space is designed well, the system can find the right things. If the space is noisy, the retriever will still return something plausible, and plausible is often the most dangerous kind of wrong. That is the entire game. * * * ## Chapter 8 — Hybrid Search Vector search is good at meaning. Keyword search is good at precision. Production needs both. That is the whole chapter. If you only use embeddings, the system understands the idea of the query but can miss the exact phrase that actually matters. If you only use keywords, the system catches exact matches but misses the intent behind the question. Hybrid search exists because real users do both things at once. * * * ## Why vector search is not enough Vector search is great when a person asks naturally. > “How do I upgrade my plan?” That kind of question has room for interpretation. The model can infer the intent even if the wording is loose. But then the user asks something like: > “Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?” Now exact words matter. * Professional. * Enterprise. * active invoices. A vector model may understand the general billing theme, but it can still miss the exact policy sentence because the answer depends on precise terms, not just conceptual similarity. That is where pure semantic retrieval starts lying politely. * * * ## Why keyword search is not enough Now flip the problem. A keyword system is brilliant when the query contains exact tokens. If the question includes: * product names, * version numbers, * error codes, * clause IDs, * policy names, * exact phrases, then BM25 or another lexical retriever often finds the right passage instantly. But if the user says: > “Can a customer move to the top tier if they still owe money?” a pure keyword search may fail because the document says: > “Active invoices must be closed before upgrading.” That is the same idea, but not the same wording. So keyword search is precise, but not smart. Vector search is smart, but not precise enough. * * * ## The answer is both Hybrid search is not some fancy optimization. It is the basic admission that no single retrieval signal is enough. The flow usually looks like this: ```plaintext Query ↓ Vector Search ↓ Keyword Search ↓ Fuse Results ↓ Rerank ↓ Send to LLM ``` The idea is simple: * semantic retrieval finds the concept, * keyword retrieval finds the exact phrase, * fusion combines the strengths, * reranking picks the best final candidates. * * * ## A real example Take this query: > “Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?” Vector search might return: * billing policy chunks, * plan upgrade chunks, * invoice-related chunks. Keyword search might return: * exact mention of “Professional”, * exact mention of “Enterprise”, * exact mention of “active invoices”. If you merge both lists, suddenly the system has a much better chance of building the full answer instead of just a vaguely related one. That is the difference between “sounds right” and “is right.” * * * ## The fusion problem The tricky part is that vector scores and BM25 scores do not live on the same scale. You cannot just add them blindly and hope the universe respects your optimism. That is why production systems use score fusion methods like: * weighted sum, * rank-based fusion, * Reciprocal Rank Fusion. The exact method matters less than the principle: do not force two different ranking systems to pretend they are the same thing. * * * ## RRF in plain English Reciprocal Rank Fusion is popular because it rewards documents that rank well in both systems without caring too much about score scale. A simple version looks like this: ```python def rrf_score(rank, k=60): return 1 / (k + rank) def fuse_rrf(vector_ranked, bm25_ranked, k=60): scores = {} for rank, item in enumerate(vector_ranked, start=1): scores[item["id"]] = scores.get(item["id"], 0) + rrf_score(rank, k) for rank, item in enumerate(bm25_ranked, start=1): scores[item["id"]] = scores.get(item["id"], 0) + rrf_score(rank, k) ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True) return ranked ``` The point is not the exact formula. The point is that the system stops trusting one retriever too much. * * * ## Metadata comes first Hybrid search should not run across everything in the universe. Before you search, you usually want to narrow the field: * language = en * document\_type = policy * version = latest * department = billing That way the retrievers are not wasting time on content that should never have been considered in the first place. This matters because a good production system is not just about finding more. It is about finding less, but better. * * * ## Example code ```python def hybrid_retrieve(query, vector_index, bm25_index, metadata_filters=None, top_k=10): vector_results = vector_index.search(query, top_k=50, filters=metadata_filters) bm25_results = bm25_index.search(query, top_k=50, filters=metadata_filters) fused = fuse_rrf(vector_results, bm25_results, k=60) top_candidate_ids = [item_id for item_id, _ in fused[:50]] return top_candidate_ids[:top_k] ``` That is the shape of the system: * search twice, * fuse, * narrow, * then rerank later. * * * ## Why hybrid search feels more natural Hybrid search works because people do not ask questions in one pure mode. Sometimes they say: * “upgrade plan” * “active invoices” * “GPT-4.1” * “ERR-5027” Sometimes they say: * “what happens if I still owe money?” * “how do I move to a higher tier?” * “does this apply to old versions?” They mix exact terms and fuzzy intent in the same sentence. Hybrid search is basically the system saying: > “Fine. I’ll handle both.” * * * ## Chapter 9 — Reranking Retrieval finds candidates. Reranking chooses the one that actually deserves to survive. That distinction sounds small until you build a real system and realize that the first-stage retriever is often good at finding the right neighborhood, but not good enough at choosing the right house. It gives you the right area. The reranker decides which door matters. * * * ## Why reranking exists A query like this: > “Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?” usually produces a decent shortlist from hybrid search. But shortlist is not answer quality. You might get: * one chunk about upgrading, * one chunk about invoices, * one chunk about billing support, * one chunk about a related policy, * and one chunk that is technically similar but not actually useful. That is normal. Retrieval is designed to cast a wide net. Reranking is what narrows that net into something the LLM can trust. * * * ## The core problem Vector search and BM25 are fast. Cross-encoder reranking is slower. So production systems split the work: ```plaintext Query ↓ Hybrid Search ↓ Top 20–50 candidates ↓ Cross-Encoder Reranker ↓ Top 3–5 chunks ↓ LLM ``` This is the standard retrieve-then-rerank shape because it balances speed and precision. The first stage optimizes recall. The second stage optimizes correctness. * * * ## Why the first ranking is not enough The first retriever often returns something that is “close enough,” which is exactly the problem. For example, imagine the shortlist contains: ```plaintext 1. "Customers can upgrade from Professional to Enterprise." 2. "Active invoices must be closed before upgrading." 3. "Billing support and payment history." 4. "How to reset your password." 5. "Downgrading is allowed only if no active trials exist." ``` A human can instantly see that 1 and 2 matter most. But the retriever only sees statistical similarity. It knows what is related, not what is most answer-bearing. That is the gap reranking closes. * * * ## How a cross-encoder thinks A cross-encoder takes the query and the candidate chunk together and scores the pair as one unit. That is different from embeddings. * A bi-encoder says: “these two texts look similar in space.” * A cross-encoder says: “this chunk answers this query better than the other chunk.” That extra interaction is expensive, but it is much more precise. This is why rerankers are usually the cheapest way to improve answer quality once retrieval is already decent. They do not fix bad retrieval. They fix bad ordering. * * * ## A simple example Suppose the query is: > “Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?” And the retrieved chunks are: * “Customers can upgrade from Professional to Enterprise.” * “Active invoices must be closed before upgrading.” * “How to reset your password.” * “Downgrading is allowed only if no active trials exist.” A reranker will likely push the first two to the top because they jointly answer the question. The third is irrelevant. The fourth is related but not the right policy branch. That is the difference between: * finding similar text, * and finding the most useful text. ## Example code Here is the shape of a basic cross-encoder reranking step: ```python from sentence_transformers import CrossEncoder reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2") def rerank(query, chunks, top_n=5): pairs = [[query, chunk["text"]] for chunk in chunks] scores = reranker.predict(pairs) ranked = sorted(zip(chunks, scores), key=lambda x: x[1], reverse=True) return [chunk for chunk, score in ranked[:top_n]] ``` That is the essential idea: * retrieve many, * score each query-chunk pair, * keep the best few. * * * ## Why reranking is so valuable Reranking improves the part of the pipeline that users actually feel. It helps when: * chunks are semantically close but not equally useful, * the corpus contains overlapping policies, * the query is specific, * exact answer selection matters, * the system has too much context noise. In practice, reranking is one of the highest ROI improvements in production RAG because it upgrades quality without forcing you to rebuild everything else. * * * ## The hidden benefit Reranking also makes debugging easier. If retrieval looks good but the answer is still wrong, the problem may be: * bad reranking, * bad chunking, * or bad context construction. If retrieval itself is weak, reranking cannot save it. That is important. Reranking is not a miracle layer. It is a refinement layer. * * * ## Where reranking sits in the architecture The right mental model is: 1. Retrieval finds enough candidates. 2. Reranking decides which candidates are worth using. 3. Generation turns those candidates into an answer. If the first stage is the net, the reranker is the hand that chooses the fish you actually keep. * * * ## The Core Idea Retrieval is not just about finding text that looks similar to a user's query. A production RAG system must determine which information is actually relevant, which sources should be trusted, and which results should be excluded before they ever reach the LLM. Vector similarity provides semantic relevance, but it does not provide enough control on its own. Hybrid search, metadata filtering, and reranking work together to narrow a large candidate set into the small amount of context that the model actually needs. That is the core idea behind production retrieval: **don't just retrieve more information—retrieve the right information, in the right order, for the right query.** * * * ## Continue the Series This article focused on the retrieval layer of a production RAG system: embeddings, hybrid search, query optimization, reranking, and context compression. Together, these techniques help turn a large set of possible matches into a smaller, more relevant context for the LLM. But a retrieval pipeline that works well on a small dataset can behave very differently when the system needs to handle millions of documents, concurrent users, strict latency requirements, and growing infrastructure costs. In the next article, we'll move from retrieval quality to production scale. We'll explore how to design RAG architectures that remain fast, reliable, and cost-efficient as the amount of data and traffic grows. **Next up:** **Part 4 — Scaling RAG Systems: Production Architecture and Performance Optimization** We'll cover: * Large-scale RAG architecture * Scaling ingestion and retrieval pipelines * Vector database performance and indexing * Caching and latency optimization * Async processing and background workers * Cost optimization * Designing RAG systems for millions of documents and concurrent users By the end of this series, you'll have a complete engineering framework for designing, building, scaling, and evaluating production-grade Retrieval-Augmented Generation systems.