RAG Retrieval: Choosing Search Methods by Quality, Speed & Cost

Updated 5 November 2026 · Standard formula, shown below

Choose RAG retrieval methods by testing answer quality, latency, and operating cost against your own questions and documents; no single search method wins on all three. Start with a simple keyword or vector baseline, then add hybrid search, reranking, or more advanced retrieval only when evaluation shows a meaningful improvement.

What retrieval does in a RAG system

Retrieval-augmented generation (RAG) gives a language model relevant passages from an external collection when answering a question. A typical system splits documents into chunks, indexes them, retrieves a shortlist for each query, and places selected text in the model’s prompt. The generator then answers using that context. Retrieval quality matters because a fluent model cannot reliably cite or use evidence it never received.

“Quality” is not just whether a search result looks relevant. A useful system must find the evidence needed for the question, avoid misleading passages, fit the context within practical limits, and support accurate answers. Speed and cost also depend on the whole pipeline: query processing, search, reranking, prompt size, and generation can all contribute.

Search methods and their trade-offs

Keyword search: Methods such as BM25 rank documents by matching query terms. They are often a strong, relatively simple baseline for exact names, identifiers, uncommon terms, and technical phrases. They can miss a passage that expresses the same idea using different words.
Vector search: An embedding model represents queries and text chunks as vectors; search finds chunks that are close in that representation. This can help with paraphrases and conceptually related wording, but may rank broadly similar passages above an exact match. Embedding generation, index storage, and vector search add their own resource requirements.
Hybrid search: This combines lexical and vector results, often by merging or reranking their rankings. It can cover both exact wording and semantic similarity, but adds configuration and evaluation work. A hybrid system is not automatically better: poorly balanced results can increase noise.
Reranking: A second-stage model examines a query and a smaller set of retrieved passages, then reorders them by estimated relevance. This can improve the top results, but adds computation and latency. It cannot recover useful evidence that the initial search did not retrieve.
Metadata filters: Filters restrict search to attributes such as document type, date, department, or access scope. They can make results more precise and reduce the search space, provided metadata is accurate and filters match the user’s request.

These methods can be combined. For example, a system might filter to current policy documents, retrieve candidates with both keyword and vector search, merge the results, and rerank the shortlist. Each stage should have a purpose that can be checked in evaluation.

How to choose: measure before adding complexity

Build a small, representative test set of real questions and the passages that contain their answers. Include varied wording, exact identifiers, ambiguous questions, and questions whose evidence is spread across documents. Keep the set separate from examples used to tune the system where practical, and update it as the collection changes.

Measure retrieval separately from generated-answer quality. For retrieval, useful measures include Recall@k—whether relevant evidence appears among the first k results—and Precision@k—how much of that shortlist is relevant. Mean reciprocal rank (MRR) rewards placing the first relevant result near the top. For end-to-end answers, assess factual correctness, evidence support, and whether citations point to the right passages. Automated scores can help compare versions, but human review is valuable for checking what a metric misses.

Establish a baseline with a consistent chunking approach and one straightforward search method.
Record retrieval metrics, answer quality, latency, and resource use on the same test queries.
Change one major factor at a time, such as search type, shortlist size, chunk size, or reranking.
Keep a change only if it improves a relevant outcome without violating your latency, cost, or operational limits.

For example, an internal support search may need exact product codes as well as natural-language questions. If keyword search finds codes reliably but misses paraphrases, hybrid search is worth testing. If the correct passages already appear in the shortlist but are buried, reranking may help. If they are absent, improve indexing, chunking, filters, or the first-stage retriever instead.

Speed and cost across the pipeline

Retrieval latency is only one part of response time. A practical measurement tracks the stages separately: embedding a query when needed, database or index search, result merging, reranking, and generation. Report typical latency as well as slower cases; averages alone can hide delays that users notice. Caching repeated queries or embeddings can help in suitable workloads, but cached results must be invalidated or refreshed when the underlying data changes.

Costs vary by system design and workload, so compare measured resource use rather than assuming that a named method has a fixed price. Relevant factors include:

How many documents and chunks must be indexed, and how often they change.
Embedding creation and storage, including reprocessing when models or documents change.
Search infrastructure, index size, query volume, and concurrent traffic.
Reranking model calls and the number of candidate passages sent to that stage.
Prompt size: adding more retrieved text may increase generation cost and can make answers less focused.

Compare cost per successful answer, not merely cost per search. A cheaper retriever that frequently misses evidence may cause retries, longer prompts, or poor answers. Conversely, extra retrieval work is difficult to justify if it does not improve outcomes users care about.

Common problems and practical fixes

A common misconception is that increasing the number of retrieved chunks always improves answers. More context can introduce irrelevant or contradictory text, consume the model’s context window, and make evidence harder to use. Tune the shortlist size using evaluation rather than choosing the largest possible value.

Relevant text is missing: Check document ingestion, chunk boundaries, metadata, and whether the query uses different wording from the source. Test an alternative retriever or hybrid search.
Results are related but not useful: Tighten filters, improve chunking, tune retrieval settings, or test reranking on candidates that already include the correct evidence.
Answers cite the wrong passage: Evaluate passage selection and citation mapping, not just the generated prose. Preserve source identifiers and verify that citations correspond to retrieved text.
Performance varies after updates: Re-run the test set after changing documents, embeddings, models, chunking, or search configuration. These changes can shift rankings.

Chunking is especially important. Very small chunks may lose context; very large chunks can dilute relevance and waste prompt space. There is no universally correct chunk size because document structure and question types differ. Preserve useful headings and boundaries where possible, then compare alternatives on the same evaluation set.

Security and reliability considerations

Retrieval can expose information if access controls are not applied consistently. Enforce permissions during retrieval, not merely by asking the language model to ignore unauthorized text. Check that metadata filters cannot be bypassed by ordinary query variations, and test with users who have different access scopes.

Retrieved documents may also contain inaccurate, outdated, or hostile instructions. Treat retrieved content as evidence, not as authority to override system rules or user permissions. Track document sources and freshness, make it possible to remove or update indexed material, and ensure answers can communicate when evidence is absent or uncertain.

Frequently Asked Questions

Is vector search always better than keyword search for RAG?
No. Vector search can help with paraphrases, while keyword search often handles exact terms and identifiers well. Test against the actual query and document mix.

When should I use hybrid search?
Test it when users ask both exact-match and meaning-based questions, or when one baseline consistently misses a class of relevant results. Keep it only if measured gains justify its added complexity.

Does reranking make retrieval slower?
Usually it adds a processing stage, so it can increase latency and resource use. Its value depends on whether better ordering of the initial candidates improves answer quality enough to matter.

How many chunks should a RAG system retrieve?
There is no universal number. Evaluate several shortlist sizes against evidence coverage, answer quality, latency, and prompt length; more chunks are not always better.

Can better search fix incorrect answers?
Only when missing or poorly ranked evidence is the cause. Incorrect answers can also result from stale documents, weak chunking, generation behavior, or faulty citations, so diagnose the pipeline stage before changing search.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.