RAG Evaluation Metrics: What Each Score Tells You

Retrieval-augmented generation (RAG) systems combine a retriever and a generator to answer questions using external knowledge. The four core evaluation metrics—faithfulness, answer relevance, context precision, and context recall—each measure a different quality: whether the answer is grounded in the retrieved context, whether it addresses the question, whether the retrieved context is concise, and whether all necessary information was retrieved. This article explains what each score means, how it is computed, and how to interpret it in practice.

Formula

Pace (min/km) = Total time (minutes) ÷ Distance (km)
Speed (km/h) = Distance (km) ÷ (Total time (minutes) ÷ 60)

Variables:
t = total time in minutes
d = distance in kilometers
Pace is expressed in minutes per kilometer; speed in kilometers per hour.

Worked Example

Using the default values: t = 10 minutes, d = 5 km.
Pace = 10 ÷ 5 = 2.00 min/km
Speed = 5 ÷ (10 ÷ 60) = 5 ÷ 0.1667 = 30.00 km/h

The calculator returns the same results: 2.00 min/km and 30.00 km/h.

Why RAG Evaluation Matters

RAG systems are only as useful as the quality of their retrieved documents and the faithfulness of their generated answers. A model that hallucinates or pulls from irrelevant passages can mislead users, especially in domains like medicine, law, or customer support. Evaluation metrics give you a quantitative way to catch these failures before deployment. The four core metrics come from the RAGAS framework, a popular open-source library, but the concepts apply to any RAG evaluation approach. Each metric targets a distinct failure mode: unfaithful answers, off-topic responses, bloated context, or missing information.

Faithfulness: Is the Answer Grounded in the Context?

Faithfulness measures the proportion of claims in the generated answer that are directly supported by the retrieved context. A high faithfulness score (close to 1) means every statement in the answer can be traced back to a source document. A low score indicates hallucination—the model invented facts not present in the context.

How it works: The evaluator (often an LLM) breaks the answer into individual claims, then checks each claim against the context. Faithfulness = (number of supported claims) ÷ (total claims). For example, if the answer makes four claims and three are supported, the score is 0.75.

Common misconception: Faithfulness does not measure whether the answer is correct—only whether it is supported by the provided context. A context could be outdated or wrong, and the answer could be faithfully wrong. Always pair faithfulness with context accuracy checks.

Answer Relevance: Does the Answer Address the Question?

Answer relevance measures how well the generated answer matches the user’s question. A high score means the answer is directly on-topic and contains no irrelevant information. A low score suggests the model answered a different question or included off-topic content.

How it works: The evaluator generates a set of questions from the answer (reverse-engineering), then computes the similarity between those generated questions and the original user question. Answer relevance = average similarity (often using embedding cosine similarity). For example, if the user asks “What is the capital of France?� and the answer says “Paris is the capital and largest city,� the generated questions would likely match the original, yielding a high score.

Limitation: This metric does not check factual correctness. An answer that is perfectly relevant but factually wrong will still score high on relevance. It only measures topical alignment.

Context Precision: Is the Retrieved Context Concise and Relevant?

Context precision evaluates whether the retrieved contexts contain only the information needed to answer the question, ranked in the correct order. A high score means the most relevant information appears early and there is little noise. A low score indicates that the retriever returned many irrelevant chunks, forcing the generator to sift through noise.

How it works: For each retrieved chunk, the evaluator checks whether it is relevant to the question. The metric uses a weighted average where chunks appearing earlier in the list get higher weight. Context precision = Σ (precision at k × relevance of chunk k) ÷ total relevant chunks. Precision at k is the proportion of relevant chunks among the first k.

Example: If the retriever returns 5 chunks, and only the 3rd chunk is relevant, the precision at k=3 is 1/3, but the score is lower than if the relevant chunk were first. This metric rewards retrievers that put the best evidence at the top.

Context Recall: Did the Retriever Find All Necessary Information?

Context recall measures whether the retrieved context contains all the information needed to answer the question correctly. A high score means the retriever captured every relevant passage. A low score indicates missing information, which often forces the generator to hallucinate or give an incomplete answer.

How it works: The evaluator compares the retrieved context against a ground-truth answer (or a set of reference contexts). It checks how many claims in the ground truth are present in the retrieved context. Context recall = (number of ground-truth claims found in context) ÷ (total ground-truth claims). For example, if the ground truth has 5 key facts and the context contains 4, the recall is 0.8.

Challenge: This metric requires a ground-truth reference, which is not always available in production. In practice, teams often use a smaller labeled set to estimate recall.

How to Choose Which Metric to Prioritize

The right metric depends on your use case and failure mode:

• Faithfulness – prioritize if hallucination is your main risk (e.g., medical advice, legal summaries).
• Answer relevance – prioritize if users complain about off-topic answers or if your model tends to ramble.
• Context precision – prioritize if your retriever returns too many irrelevant chunks, slowing down generation or confusing the model.
• Context recall – prioritize if users report missing details or if the retriever often fails to find key documents.

In practice, you should track all four, but weight them according to your product’s priorities. A chatbot for customer support might value faithfulness and answer relevance most, while a research assistant might value context recall to ensure comprehensive answers.

Common Mistakes and Misconceptions

• Treating scores as absolute truth – All four metrics are LLM-based or embedding-based approximations. They are noisy and can be biased. Always validate with a small human-annotated set.
• Ignoring the ground truth requirement – Context recall needs a reference answer. Without it, you can’t measure recall reliably.
• Using only one metric – A system can score high on faithfulness but low on relevance, or vice versa. The four metrics are complementary.
• Not normalizing for context length – Longer contexts can artificially inflate recall (more chances to contain the answer) but hurt precision. Always report both.
• Assuming higher is always better – A perfect score on all four is rare. In practice, trade-offs exist: maximizing recall may reduce precision, and maximizing faithfulness may make answers overly conservative.

Frequently Asked Questions

What is a good RAG evaluation score?

There is no universal threshold. Scores above 0.8 are generally considered strong for faithfulness and answer relevance, while context precision and recall often range between 0.5 and 0.9 depending on the difficulty of the retrieval task. Always compare scores against a baseline (e.g., a simple BM25 retriever) rather than using absolute numbers.

Can I use these metrics without a ground truth?

Faithfulness and answer relevance do not require a ground truth—they compare the answer to the context and the question. Context precision also works without ground truth (it uses the question and context). Only context recall requires a reference answer. For production systems without labels, you can compute the first three and use a small labeled set for recall.

How do these metrics compare to human evaluation?

Human evaluation is more reliable but slower and costlier. LLM-based metrics correlate reasonably well with human judgments for English text, but they can fail on nuanced or domain-specific language. A common practice is to use automated metrics for rapid iteration and then validate a sample with human annotators before release.

Are these metrics suitable for non-English languages?

They can be used, but the quality depends on the underlying LLM or embedding model. For low-resource languages, the metrics may be less accurate. It is advisable to test on a small set of examples in your target language to verify that the scores align with human intuition.

Can I use these metrics to improve my RAG system?

Yes. If faithfulness is low, try better prompts or a more capable generator. If context precision is low, improve your chunking strategy or rerank results. If context recall is low, expand your retrieval top-k or use hybrid search. The metrics tell you which component to fix.

Related

• Wi-Fi Channel Width & Auto Channel Selection Explained
• Google Pixel 8 Renders Leaked? What We Know
• Stone Calculation per Square Foot

in t | Total time (min) | min | 10 | 0
in d | Distance (km) | km | 5 | 0
out pace | Average pace (min/km) | min/km | t / d | 2
out speed | Average speed (km/h) | km/h | d / (t / 60) | 2