Build a Python RAG System for Document Questions

To build a Python RAG system for document questions, extract text from your files, split it into searchable chunks, retrieve the most relevant chunks for each question, and give those passages to a language model to answer from. A practical system also tracks source locations, handles questions it cannot support, and tests retrieval separately from answer quality.

What a document RAG system does

RAG means retrieval-augmented generation. Rather than asking a model to rely only on information it learned during training, a RAG application searches a collection of documents at question time and supplies selected passages as context. For example, when asked, “What is the expense-report deadline?”, it might retrieve a paragraph from an employee handbook and generate an answer with the handbook’s page or section as a citation.

This approach is useful when information is private, changes over time, or is too large to include in every prompt. It does not train the model on the documents: the source files remain in your retrieval system, and the model receives relevant excerpts for a particular request. RAG can reduce unsupported answers, but it does not guarantee correctness. Bad extraction, weak search, or misleading source material can still produce a bad response.

Core components and choices

A basic implementation has five stages: ingestion, chunking, indexing, retrieval, and answer generation. Start with a small set of representative files and build each stage so you can inspect its output.

• Ingestion: Read text files, PDFs, word-processing files, or HTML with suitable parsers. Scanned pages need optical character recognition (OCR); ordinary PDF text extraction will not reliably read text that exists only as an image.
• Chunking: Divide extracted text into passages small enough to retrieve and fit into the model’s context. Preserve useful boundaries such as headings and paragraphs when possible.
• Indexing: Store each passage, its metadata, and a searchable representation. Metadata commonly includes filename, page number, heading, and access-control scope.
• Retrieval: Find passages relevant to the question using keyword search, vector similarity, or a combination.
• Generation: Provide the question and selected passages to a language model with instructions to answer from evidence and identify sources.

For a small prototype, local files and a lightweight vector index may be enough. Larger or multi-user systems often need a persistent database, incremental indexing, permission-aware search, and monitoring. Choose components based on data size, update frequency, privacy requirements, and operational skills—not just benchmark scores.

Build the pipeline in Python

Python libraries can handle document parsing, embeddings, vector storage, and model calls, but the stages are the same whether you use a framework or assemble them directly. A minimal design can be expressed as the following pseudocode; the exact library calls depend on your selected providers and file formats:

documents = load_documents("./docs")
chunks = split_documents(documents, chunk_size=800, overlap=120)
for chunk in chunks:
    chunk.embedding = embed(chunk.text)
index.upsert(chunks)

def answer(question):
    matches = index.search(embed(question), top_k=5)
    context = format_with_sources(matches)
    return generate(
        question=question,
        context=context,
        instruction="Answer from the supplied sources; say when they do not support an answer."
    )

The numbers above are starting settings, not universal best values. Chunk sizes may be measured in characters, tokens, or another unit depending on the implementation. Verify the definition before tuning. Overlap can preserve context across chunk boundaries, but excessive overlap creates duplicate results and wastes storage and prompt space.

Keep the original text and metadata alongside each chunk. Assign stable document and chunk identifiers so an updated file can replace old content instead of accumulating stale copies. For PDFs, attach page numbers where the parser provides them; for other formats, store headings or section paths. These details make citations useful and help debug retrieval.

Improve retrieval and answer quality

Vector search represents text as embeddings and compares the question’s vector with stored passage vectors. It can find conceptually similar wording even when the exact terms differ. Keyword search is often stronger for exact identifiers, names, dates, and codes. Hybrid retrieval combines both signals; a reranker can then reorder a shortlist using a more detailed relevance model.

Do not assume that returning more passages always improves answers. Too many weak matches can distract the model, while too few may omit a necessary qualification. Inspect retrieved passages for real questions from your users, and tune chunk boundaries, the number of results, and any score threshold against those examples. If documents contain tables, preserve their row and column relationships rather than flattening them into ambiguous text.

Prompt instructions should distinguish evidence from instructions embedded in documents. A retrieved passage is untrusted content: it might contain text that attempts to direct the model. Tell the model to treat retrieved material as source data, not as system instructions, and do not let document text trigger tools or privileged actions without separate validation. Require citations that point only to retrieved sources, then check that the cited passage actually supports the claim.

Testing, operations, and common failures

Evaluate the search and the generated answer separately. Create a small, representative question set with expected source documents or passages and acceptable answers. Retrieval tests reveal whether the needed evidence appears in the top results; answer tests reveal whether the model uses that evidence accurately and admits when it is missing.

• Empty or poor answers: Check extraction first, then inspect chunk boundaries and retrieved results. A language model cannot cite text that never entered the index.
• Confident but unsupported claims: Strengthen evidence-based instructions, limit context to relevant passages, and test an explicit “not found” response.
• Outdated information: Re-index changed files, remove deleted content, and record when each source was ingested. A vector index is not automatically synchronized with its source folder.
• Broken citations: Store page or section metadata at ingestion and carry it through retrieval and response formatting.
• Slow or costly requests: Measure parsing, embedding, retrieval, and generation separately. Cache or batch repeated work where appropriate, and avoid embedding unchanged documents again.

For sensitive collections, apply access controls before retrieval or filter results by the requesting user’s permissions. Keeping access checks only in the prompt is not sufficient: restricted passages should not be sent to a model in the first place. Review where document text, embeddings, logs, and prompts are stored, and set retention and deletion behavior deliberately.

Frequently Asked Questions

Do I need a vector database for a Python RAG system?
A small collection can use an in-process index or even keyword search. A dedicated persistent vector store becomes useful when you need larger-scale search, updates, metadata filters, or multiple application instances.

Can RAG answer questions about scanned PDFs?
Yes, if you first run OCR and check its output. Poor scans, complex layouts, and tables can lead to extraction errors, so validate representative pages before indexing a large collection.

How large should RAG chunks be?
There is no single correct size. Start with passages that preserve a coherent idea, then test retrieval on real questions. Adjust size and overlap based on whether results omit context or include too much unrelated text.

Does RAG prevent hallucinations?
No. It gives the model relevant evidence, but retrieval can fail and the model can misread or overstate a passage. Test for unsupported answers and provide a clear way to say that the documents do not contain enough information.

Should documents be embedded every time someone asks a question?
Usually not. Embed and index documents during ingestion, then embed each incoming question for search. Reprocess only new or changed content, while removing obsolete chunks when a source is replaced or deleted.

Related

• SMBs: Build Repeatable ROI From Everyday AI Workflows
• How Many Blocks for a Full Beacon in Minecraft?
• Change Google Calendar Event Colors on Any Device