RAG: How LLMs Answer Questions Using Your Data
Retrieval-augmented generation (RAG) is a method for letting a large language model answer questions using information retrieved from a selected collection of documents. A RAG calculator would not meaningfully measure this process, so this page explains what RAG is, how it works, and what to consider when using it with your data.
What RAG means
RAG stands for retrieval-augmented generation. It is a technical term, not an informal expression or a separate kind of language model. “Retrieval” means finding relevant information; “augmented” means adding that information to the model’s input; and “generation” is the model’s production of a response.
The term was introduced in a 2020 research paper, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” by Patrick Lewis and coauthors. The paper described a system combining a language model with a searchable external knowledge source. In practical use, people often say “RAG” for both this general approach and the software pipeline built around it.
For example: “We added RAG to the support assistant so it can answer questions from the product manuals.” The sentence describes a system that looks up relevant manual passages and gives them to a language model when answering. RAG does not mean that the model has permanently learned the manuals.
How a RAG system works
A basic RAG system has two stages: prepare and search the information, then use retrieved passages to answer a question. Implementations vary, but a typical workflow is:
- Collect and prepare sources. Documents such as web pages, PDFs, policies, or database records are extracted into text. The system may remove repeated headers, preserve titles, and attach metadata such as dates, departments, or access permissions.
- Split the material into chunks. Long documents are divided into smaller sections so search can return focused passages rather than an entire file. Chunk size and overlap affect whether an answer’s context stays together; there is no universally best setting.
- Create searchable representations. A common approach converts each chunk into an embedding, a numerical representation used to find text with related meaning. The vectors and source text are stored in a search index. Some systems also use keyword search, metadata filters, or a combination.
- Retrieve passages for a question. The user’s query is searched against that index. The system selects a limited set of passages, sometimes reorders them with a reranking step, and may apply permission filters.
- Generate a response. The selected text and the question are placed in the model’s prompt. Instructions may ask the model to answer from those sources, cite them, or say when the evidence is insufficient.
For example, if an employee asks, “How many days do I have to submit an expense report?”, retrieval might return the relevant policy paragraph. The model then summarizes that passage. The answer is only as dependable as the retrieved text, the prompt, and the model’s interpretation of it.
RAG versus model training
RAG is often confused with fine-tuning or training. They solve different problems. RAG supplies information at question time; it can make an existing model consult changing or specialized documents without incorporating every document into its model parameters. Fine-tuning adjusts a model using examples to influence its behavior or response style. It is not, by itself, a dependable way to make a model retrieve current facts from a large, frequently changing document collection.
RAG does not necessarily update a model’s underlying knowledge, and it does not guarantee factual answers. A model can misunderstand a source, combine passages incorrectly, or produce a claim that the sources do not support. It is better thought of as a way to provide relevant evidence than as a truth-checking mechanism.
What RAG is useful for
RAG is useful when answers should draw on a known body of material, especially if that material changes or is too large to include in every prompt. Typical applications include internal policy assistants, product documentation search, research over a defined collection, and question-answering over a set of reports.
It may be unnecessary for general questions that a model can already answer, or for tasks where a conventional search result or database query is more reliable. If the answer must be an exact count, current inventory value, or other structured value, a direct query to the authoritative system may be preferable to asking a language model to infer it from text.
RAG also has practical limits. The model has a finite context window, so only some retrieved material can be supplied at once. Indexing can become outdated when source documents change. Scanned or poorly formatted files may not extract cleanly, and ambiguous questions can retrieve the wrong passages.
Common problems and ways to improve results
A polished answer can still be wrong. Useful checks focus on both retrieval and generation:
- Relevant evidence is missing: Check extraction, chunk boundaries, search terms, and metadata filters. Compare results for real user questions, not just ideal examples.
- Search returns similar but incorrect text: Combine semantic retrieval with keyword matching where appropriate, and test whether a reranking step improves the top results.
- The answer goes beyond its sources: Ask for citations tied to retrieved passages and instruct the model to acknowledge when the evidence does not answer the question. Verify citations; a citation label alone does not prove support.
- Information is stale: Establish a process to re-index changed or deleted documents and confirm that search no longer returns obsolete versions.
- Answers vary: Keep prompts and retrieval settings consistent, test repeated and difficult queries, and use a human review process for consequential decisions.
Evaluation should test whether the right passages are retrieved as well as whether the final answer is correct and supported. Looking only at fluent answers can hide retrieval failures.
Privacy, permissions, and security
Connecting a model to private files does not automatically make those files safe. A RAG service must enforce who may search and receive each document. Permissions should be applied during retrieval, not merely described in a prompt, because a model cannot reliably enforce access control through instructions alone.
Consider what data is sent to the model provider, how prompts and outputs are logged, how long indexes and embeddings are retained, and how deletion requests propagate. Embeddings are derived from source content and should not be casually treated as anonymous. Restrict access to indexes, protect credentials, and test that one user cannot retrieve another user’s restricted material.
Retrieved documents can also contain misleading instructions or hostile text. Treat document contents as data to analyze rather than trusted system instructions, and use safeguards appropriate to the application. For sensitive or high-impact use, include human oversight and verify answers against authoritative sources.
Frequently Asked Questions
Does RAG train an LLM on my documents?
No. In a typical RAG setup, documents are indexed and relevant passages are provided in the prompt when a question is asked. That is different from changing the model’s parameters through training or fine-tuning.
Can RAG stop hallucinations?
No. It can provide evidence that may reduce unsupported answers, but retrieval can fail and a model can misread or ignore the evidence. Test answers and check important claims against their cited sources.
Does RAG work with PDFs?
Often, if the text can be extracted or recognized reliably. Scanned pages, complex tables, columns, and figures may need optical character recognition or specialized processing, and extraction errors can affect retrieval.
Is RAG the same as a vector database?
No. A vector database is one possible component for storing and searching embeddings. RAG is the broader process of retrieving relevant material and using it to help generate an answer; systems can also use keyword or hybrid search.
Can a RAG assistant cite its sources?
It can be designed to show the documents or passages used, but citations need validation. Check that each cited passage exists and actually supports the claim beside it.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.