Hybrid RAG: Neo4j Graph Memory + Vector Search
Building a hybrid RAG system that combines Neo4j graph memory with vector search gives you retrieval that understands both semantic similarity and structured relationships. You get the best of both worlds: dense vector embeddings for fuzzy matching and a knowledge graph for precise, multi-hop reasoning and factual consistency.
Why Combine Graph and Vector Retrieval?
Standard RAG pipelines use vector similarity to fetch chunks of text. This works well for finding passages that are semantically close to a query, but it struggles with questions that require joining information across multiple documents or understanding relationships like “who worked with X on project Y?” A vector index treats each chunk as an isolated point in high-dimensional space, so it cannot represent edges, hierarchies, or shared entities. A graph database like Neo4j stores entities as nodes and relationships as edges, making it natural to answer questions like “list all employees who report to Alice and have a skill in Python.” However, graphs alone are brittle: they require exact entity names and structured queries, and they fail on vague or paraphrased questions. Hybrid RAG addresses this by using vectors to find relevant starting points and then traversing the graph to gather connected context.
How the Architecture Works
A typical hybrid setup has three stages: ingestion, retrieval, and synthesis. During ingestion, you process documents into chunks, generate embeddings for each chunk, and also extract entities and relationships to populate the Neo4j graph. The graph nodes can represent chunks, entities, or both, and edges can be semantic relationships like “mentions” or “part_of.” At query time, you run two parallel retrievals: a vector similarity search to get the top-k chunks, and a graph query that may use entity extraction from the query to find related nodes. You then merge the results, deduplicate, and rank them before feeding them to the LLM. A common pattern is to use the vector results to identify relevant entities, then expand those entities in the graph to pull in neighboring nodes that provide additional context—this is called graph-based context expansion.
Choosing the Right Graph Schema
Your schema determines what kinds of questions your system can answer. Start by listing the entity types that matter for your domain—people, places, products, events—and the relationships between them. For a document corpus, a simple schema is: Chunk nodes with embedding properties, Entity nodes, and a MENTIONS relationship from Chunk to Entity. You can also add SOURCE nodes to track provenance. Avoid over-normalizing; you want a schema that is easy to query and update. For example, if you are building a support bot, you might have Issue, Solution, and Product nodes with RELATES_TO and FIXES edges. A good practice is to design your schema around the top 10 questions you expect to answer.
Implementing the Hybrid Retrieval Pipeline
Here is a concrete approach using Neo4j and a vector index (e.g., with the neo4j Python driver and an embedding model):
- Store each chunk as a node with a text property and an embedding property. Create a vector index on that property.
- Extract entities from each chunk using an NLP library or an LLM, and create/merge entity nodes. Create MENTIONS relationships with a weight or context property.
- At query time, embed the user question and run a vector similarity search to get top-k chunks (e.g., k=5).
- Parse the question to extract candidate entities (using a small LLM or a NER model).
- Run a Cypher query that finds those entities and their neighbors up to depth 2, collecting the connected chunks.
- Merge the two result sets, score them (e.g., weighted sum of vector similarity and graph proximity), and take the top-n.
- Feed the combined context to the LLM with the original question.
This pipeline can be implemented as a Python service using the Neo4j driver and a vector store like FAISS or the native Neo4j vector index (available in Neo4j 5.11+).
Common Pitfalls and How to Avoid Them
One major pitfall is inconsistent entity resolution: the same entity may appear as “NYC” and “New York City,” causing the graph to fragment. Use a normalization step (lowercasing, alias mapping) and a resolution service to merge duplicates. Another issue is that vector and graph results can be redundant; you need a deduplication strategy that keeps the most informative chunks. Also, beware of query latency: deep graph traversals can be slow on large graphs, so limit depth and use indexes on relationship types. Finally, do not ignore the quality of your embeddings—a poorly tuned embedding model will degrade both retrieval paths. Test with a domain-specific evaluation set.
Cost and Performance Considerations
Running a hybrid system involves costs for the graph database (memory and storage), the embedding model (if using an API), and the LLM for synthesis. Neo4j Community Edition is free but has limitations on clustering; Enterprise adds horizontal scaling. Vector indexes in Neo4j are in-memory, so you need enough RAM to hold embeddings. To control costs, you can cache frequent query results and use a smaller embedding model for entity extraction. Performance-wise, the main bottleneck is often the graph traversal, so use parameterized queries and set timeouts. Benchmark your system with a realistic query set to measure p95 latency.
Alternatives to Neo4j for Graph Memory
Neo4j is the most popular graph database, but other options exist. Memgraph is a drop-in replacement that is in-memory and faster for some workloads. Amazon Neptune is a managed service that supports both property graphs and RDF. If you want a lighter solution, you can store adjacency lists in a relational database or even in a JSON blob, but you lose the query power of Cypher. For very large graphs, consider using a graph database with distributed support like JanusGraph. The choice depends on your scale, budget, and whether you need ACID transactions.
Frequently Asked Questions
Do I need a graph database at all, or can I just use vector search?
If your queries are simple factoid lookups, vector search alone may suffice. But if you need to answer questions that require joining multiple pieces of information or reasoning over relationships, a graph provides structure that vectors cannot. A hybrid approach is more robust and future-proof.
How do I keep the graph and vector index in sync when documents change?
Use a change-data-capture mechanism: whenever a document is added, updated, or removed, trigger a pipeline that updates both the chunk nodes and the entity relationships. For deletions, you must remove all edges connected to the deleted entities. This can be done with a message queue or a scheduled job.
What embedding model should I use for the vector search?
Choose a model that matches your domain and language. Sentence-transformers (e.g., all-MiniLM-L6-v2) are a good default for English. For multilingual or specialized domains, consider models like BGE or E5. Evaluate on a small set of representative queries to see which gives the best retrieval quality.
Can I use an LLM to extract entities from documents automatically?
Yes, using an LLM for entity extraction can be more accurate than traditional NER, especially for complex or domain-specific entities. However, it is slower and more expensive. A practical approach is to use a lightweight NER model for initial extraction and an LLM for disambiguation and relationship extraction.
How do I evaluate the performance of a hybrid RAG system?
Build a test set of questions with known correct answers. Measure retrieval recall (are the relevant chunks in the top-n?) and answer accuracy (does the LLM produce the right answer?). You can also measure the number of hops needed to answer a question, which indicates the value of the graph component. Use metrics like hit rate and mean reciprocal rank.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.