How to Evaluate RAG Systems: Metrics, Methods, and What to Measure First

The material is prepared for future students of the course "AI-Architect".

When a RAG system fails, it is impossible to understand why it happened based solely on the answer. RAG stands for retrieval-augmented generation – generation augmented by search – and it is one of the most common techniques for context design, allowing AI agents to be supplemented with additional information, thus improving their performance. Since RAG is a critical component of modern AI applications, developers need an LLM evaluation method that allows them to identify problems and track RAG performance quality.

This guide will explain what exactly needs to be measured at each stage of the RAG pipeline, why each metric is important, and how to structure the evaluation process so that it not only identifies problems but also shows where exactly they occur.

We will also discuss the most effective evaluation approach – LLM-as-a-judge. Connecting LLM to the evaluation stage has largely replaced outdated deterministic metrics like BLEU and ROUGE, which measured word overlap rather than semantic accuracy. LLMs are better at assessing textual relevance, making them a good fit for the RAG evaluation toolkit.

How RAG Systems Fail and Why Evaluation Should Be Separate

Failures in RAG systems are divided into three categories that may appear the same externally, but require entirely different methods of correction:

• The search module may fail to find relevant documents or rank them too low, resulting in the model lacking the necessary context.

• The model may hallucinate, i.e., generate a fabricated response, even if relevant documents were found.

• The model may answer a different question than the one asked by the user, even if its response is well-supported by the retrieved context.

Externally, all of these failures look the same because the result is the same – an incorrect answer. However, the way to fix each case will be completely different. This is the key diagnostic problem when evaluating RAG. If you evaluate only the final answer, you reduce different types of failures to a single opaque signal.

Let's consider a specific example. The user asks the internal HR assistant: "Do I have the right to parental leave if I have been working here for 11 months?" The system responds: "Employees are entitled to 12 weeks of parental leave." Was this a search error, where the system found a general leave policy document but missed the document with the conditions stating a minimum 12-month tenure? Or was it a generation error, where the necessary document was found, but the model ignored the tenure requirement and only answered the surface-level part of the question?

These cases need to be corrected differently. A search error could mean that when splitting the information into fragments, the criteria for leave eligibility got separated from the benefits details. A generation error could mean that your prompt lacks an explicit instruction to check prerequisites before responding. Separate evaluation—where the search module and generator are checked independently of each other, and then their combined work is evaluated—helps localize weak points and make targeted improvements instead of guessing blindly.

Three Dimensions of RAG Quality: The "RAG Triad"

The most useful diagnostic scheme for assessing RAG measures three interrelated connections:

• Relevance of the found context: the connection between the user's query and what the search module found.

• Fidelity to the context, meaning reliance on the source: the connection between the found context and what the model generated.

• Relevance of the response: the connection between the user's original prompt and the final answer, i.e., did the system answer the question or fulfill the user's task?

It is easiest to think of this as three questions you ask for each interaction session: did the search module find relevant information? Did the generator stick to the facts provided? Did the final answer truly answer the user's question?

Since failures can occur on both the search and generation sides, the most useful RAG evaluation strategies involve separate checks of the search module and the generator, and then measuring the quality of the entire system from start to finish.

LLM-as-a-Judge: The Evaluation Mechanism Behind These Metrics

Most of the metrics discussed in this guide are based on the LLM-as-a-judge approach: when a sufficiently powerful LLM is used to evaluate the results produced by other models. This approach has largely replaced outdated token-overlap-based metrics, such as BLEU and ROUGE. These metrics measured word overlap on the surface level of the text, rather than meaning or factual accuracy. A response that conveys the correct information but in different words would receive a low score on BLEU; an LLM in the role of a judge is capable of recognizing semantic equivalence.

It’s worth knowing about two frameworks. G-Eval (Liu et al., EMNLP 2023) suggested using step-by-step reasoning for evaluation, where the judge model first formulates the criteria step by step before giving a score and then uses token probabilities to generate continuous rather than integer scores. In Opik, G-Eval is implemented as an embedded metric, so it can be directly applied to the results of your RAG system. Prometheus (Kim et al., ICLR 2024) showed that open-source models, fine-tuned specifically for evaluation tasks, can demonstrate the same correlation with human evaluations as GPT-4 if they are given structured rubrics. For RAG evaluation pipelines, this is important because you’re not tied to proprietary models in the role of a judge: platforms like Opik allow you to use any LLM supported by LiteLLM, including open-source options, as the evaluation basis.

Evaluating the RAG Triad in Practice

Let’s take a closer look at each dimension of the RAG triad: what exactly it measures, why it’s important, and how to implement such evaluation using Opik.

How to measure search quality in RAG systems?

The first dimension evaluates the connection between the user query and the retrieved context. Context relevance shows how well the retrieved documents actually relate to what the user asked about. If the search module retrieves irrelevant fragments, two things happen: you waste tokens in the LLM’s context window and increase the risk that the model will include unnecessary noise in the response.

In this measurement, context precision shows how well the retrieval module ranks relevant information. Finding the correct document is important, but equally important is the position it occupies in the list. The study by Liu et al. showed that when processing long contexts, language models exhibit a U-shaped quality curve: they handle information well at the beginning and end of the input but often miss what is in the middle. Because of this "middle loss" effect, a retrieval module that hides the most relevant document in 8th place out of 10 is essentially almost as useless as if it hadn't found it at all.

Opik provides both ContextPrecision and ContextRecall as built-in metrics based on the LLM-as-a-judge approach. Both use few-shot prompting with structured rubrics and score on a scale from 0.0 to 1.0:

from opik.evaluation.metrics import ContextPrecision, ContextRecall

precision_metric = ContextPrecision()
recall_metric = ContextRecall()

precision_score = precision_metric.score(
input="Am I eligible for parental leave if I've been here 11 months?",
output="Employees are eligible for 12 weeks of parental leave.",
expected_output="Employees must complete 12 months of tenure to qualify.",
context=["Parental leave policy: Eligible employees receive 12 weeks…",
"Eligibility: Employees must complete 12 months of continuous employment."],
)

recall_score = recall_metric.score(
input="Am I eligible for parental leave if I've been here 11 months?",
output="Employees are eligible for 12 weeks of parental leave.",
expected_output="Employees must complete 12 months of tenure to qualify.",
context=["Parental leave policy: Eligible employees receive 12 weeks…",
"Eligibility: Employees must complete 12 months of continuous employment."],
)

By default, both metrics use GPT-4o as the evaluator, but if necessary, you can switch to any model supported by LiteLLM by setting the model parameter. The ContextPrecision metric penalizes systems that place relevant documents lower in the ranked list, whereas ContextRecall shows whether the retrieval module extracted all the relevant information needed for the expected answer.

How to detect hallucinations in RAG responses?

The second side of the triad, groundedness, that is, substantiation – in the Ragas framework it is also called faithfulness to context – evaluates whether the generator's answer is actually supported by the retrieved context. This is your main tool for detecting hallucinations, and for most product teams, this metric turns out to be the most important in the entire evaluation stack.

Typically, context faithfulness scoring works like this under the hood. The system first breaks the generated answer into individual factual statements. For example, the answer “You are entitled to 12 weeks of parental leave after your first year of work, and you can split it into two periods” contains three statements: the duration of leave is 12 weeks, it requires one year of service, and it can be split into two parts. Then the evaluation checks each statement against the retrieved context. If the policy document says nothing about splitting the leave, then the third statement is a hallucination: the model added a plausible detail from its training data.

Context faithfulness is the ratio of confirmed statements to the total number of statements. A score of 1.0 means every statement can be traced back to the retrieved context. A score of 0.6 means 40% of the statements came from somewhere else — most likely from the model's training data or were entirely fabricated. High context faithfulness values indicate that the generator behaves like a “natural language layer” over your knowledge base rather than improvising from its parametric memory.

In Opik, hallucination detection is implemented as a built-in metric:

from opik.evaluation.metrics import Hallucination

metric = Hallucination()

score = metric.score(
input="Am I eligible for parental leave if I've been here 11 months?",
output="You're eligible for 12 weeks of parental leave after your first year, and you can split it into two blocks.",
context=["Eligible employees receive 12 weeks of parental leave.",
"Employees must complete 12 months of continuous employment to qualify."],
)

Low scores are a warning signal: the model is adding information you did not provide. This is especially dangerous in areas such as healthcare, law, or financial services, where accuracy is non-negotiable.

What measures the relevance of an answer when evaluating RAG?

The third party of the triad reveals a subtle but important type of failure: the answer may be factually justified and technically correct, but it does not address the user's question in substance. The relevance of the answer indicates how well the generated response aligns with the original query and lowers the score for answers that are incomplete, excessive, or go off-topic.

To compute the relevance of an answer without a pre-written benchmark answer, a "reverse recovery" approach can be used. The LLM generates several hypothetical questions that the current answer could realistically respond to, and then the system measures the semantic similarity between these synthetic questions and the actual user query. If the answer is indeed relevant, then the reconstructed questions should be very close to what the user originally asked.

This metric captures scenarios that are not revealed by contextual correctness alone. The user asks, "How do I reset my password?" And the system responds with a detailed, well-reasoned explanation of your company's security architecture. Each statement is supported by found context. The assessment of contextual correctness is perfect. But the user still does not understand how to reset the password. The answer relevance metric penalizes precisely this kind of gap.

In Opik, there is a metric called AnswerRelevance that implements this approach. Combined with ContextPrecision, ContextRecall, and Hallucination, these four metrics provide complete diagnostic coverage of the RAG triad.

Search metrics for tuning in production

The RAG triad gives you high-level diagnostics of the system's state. But when fine-tuning the actual configuration of the search module – including parameters such as fragment size, top-K settings, and embedding model selection – more detailed metrics from classical information retrieval theory will be needed.

Recall@K shows the proportion of all relevant documents that appeared in the top K results. If your knowledge base contains five documents capable of answering a specific query, and the search module returns only three of them in the top ten, then recall@10 will be 0.6. This metric is especially important in areas such as law or medical research, where missing even a single relevant document could lead to an incomplete or incorrect conclusion.

Precision@K measures the opposite: what proportion of the top K results is actually relevant. If you retrieved 10 documents, but only 3 are useful, then precision@10 is 0.3. Low precision means you are filling the LLM's context window with unnecessary noise, wasting tokens, and increasing the likelihood that the model will get confused by irrelevant content.

Mean Reciprocal Rank, or MRR, focuses on how quickly the search module finds the single best result. This metric calculates the average of the reciprocal rank of the first relevant document over a set of queries. An MRR of 1.0 means that the best document is always ranked first. An MRR of 0.5 means that it usually appears second. This metric is especially important in cases where the system typically needs a single authoritative answer, rather than synthesizing information from multiple sources.

Normalized Discounted Cumulative Gain, or NDCG, takes graded relevance into account, recognizing that some documents may be more relevant than others, rather than considering relevance as a binary characteristic. This metric rewards systems that place the most relevant documents at the top by applying logarithmic discounting to the contribution of results lower in the list. For complex queries, where different documents contribute different parts of the answer, NDCG provides a more nuanced picture than binary precision or recall.

These metrics also help optimize search hyperparameters. If you have a low contextual relevance score, meaning the proportion of truly useful text among the retrieved fragments is small, you may be extracting too large chunks of text. When only a small part of each fragment is relevant to the query, it’s a signal to experiment with smaller fragment sizes or a more aggressive re-ranking stage.

Evaluator Biases to Watch Out For

LLMs acting as evaluators carry systematic biases that can distort your assessments. The most common ones are positional bias, where preference is given to answers based on their position in the prompt rather than their quality; verbosity bias, where longer answers receive higher scores regardless of content; and agreement bias, where the model is better at confirming correct answers than identifying incorrect ones. Because of this, automatic evaluation can systematically overestimate system reliability.

Specifically for RAG evaluation, it is worth knowing about a calibration approach called Prediction-Powered Inference, or PPI, proposed in the ARES framework (Saad-Falcon et al., 2024). PPI uses a small set of human-labeled examples to statistically adjust automatic scores and add confidence intervals. It is a kind of reality check that helps calibrate the level of trust in all other results from the automatic evaluator.

How to Implement RAG Evaluation in Practice

Now that the metrics are clear, let’s discuss how to turn their use into a reproducible workflow.

Start Collecting an Evaluation Dataset as Early as Possible

The foundation of any evaluation pipeline is a test dataset of “query–response” pairs that reflects the intended usage scenarios of your system. Start collecting it before optimizing anything else. A useful evaluation dataset should include different types of queries: simple factual questions with unambiguous answers, complex questions requiring synthesis of information from multiple documents, ambiguous queries where the system needs to handle uncertainty correctly, and “negative” queries that the system should refuse to answer because the required information is not in the knowledge base.

You don't need thousands of examples to get started. Even 50–100 well-constructed "prompt–response" pairs covering the key scenarios your system should handle will give you a meaningful foundation. Gold-standard examples, annotated by experts, are ideal for testing in high-risk scenarios, but they are expensive to prepare.

Scale with synthetic data, but carefully

To supplement human-annotated datasets, you can use an LLM to generate synthetic test data based on your document corpus. A common approach looks like this: you feed documents into a sufficiently strong model and ask it to generate several diverse questions and corresponding answers based on their content. This is useful for rapid iterations and expanding coverage, but there is an important caveat: synthetic data reflects the understanding of the generating model, not necessarily the true state of affairs.

For production systems in high-risk areas—such as healthcare, finance, or legal services—synthetic data should be considered only as a starting point for testing during the development phase. Before making decisions to deploy the system to production, results should always be checked against "gold" sets that have undergone human verification.

Evaluation with Opik: end-to-end workflow

The strongest teams treat RAG evaluation the same way development teams treat unit testing: it runs automatically, blocks deployment when quality drops, and produces results that are understandable by the whole team. Here’s how it works in Opik.

First, define your metrics. Start with coverage of the RAG triad: Hallucination, ContextPrecision, ContextRecall, and AnswerRelevance. The evaluate function in Opik accepts a list of metrics and runs them all on your dataset in a single pass:

from opik.evaluation import evaluate
from opik.evaluation.metrics import (
Hallucination, ContextPrecision, ContextRecall, AnswerRelevance
)

metrics = [
Hallucination(),
ContextPrecision(),
ContextRecall(),
AnswerRelevance(),
]

results = evaluate(
dataset=your_dataset,
task=your_rag_pipeline,
scoring_metrics=metrics,
experiment_config={
"model": "gpt-4o",
"chunk_size": 512,
"top_k": 5,
},
)

Set pass and fail thresholds. You need to explicitly define what is considered “good enough” quality for your scenario – for example, context fidelity should be above 0.85, and answer relevance above 0.75. Run the evaluation as part of your CI/CD pipeline so that changes in the prompt or search configuration updates that cause regression are caught before going into production.

Compare experiments. The experiment_config parameter allows you to label each evaluation run with the configuration that generated it: model, chunk size, top-K value, prompt version. Then the Opik interface allows you to compare experiments side by side, so you can clearly see how configuration changes affected each metric.

Move to monitoring in production. Once the system is running in live mode, Opik’s Online Evaluation rules allow you to automatically run the same metrics on production traces. If the context fidelity drops, you can drill down into the specific trace that scored low, see which documents were retrieved, examine the prompt sent to the LLM, and understand where the failure occurred – during retrieval or generation.

This is exactly where observability and quality evaluation converge. Logging traces during development helps iterate faster. Logging in production helps detect drift and degradation. And automatic evaluation of these traces turns raw observability data into actionable signals about system quality.

Stress Testing and Adversarial Evaluation

The metrics discussed earlier evaluate whether the RAG system works correctly under normal conditions. But product systems must also handle other types of input: ambiguous, malicious, or specifically crafted to exploit pipeline vulnerabilities. Stress testing and adversarial evaluation allow you to check how the system behaves when something goes intentionally wrong.

Boundary Testing: What Happens Beyond the "Successful Path"?

Before considering competitive attacks, it's worth checking how the system handles legitimate but challenging inputs. These include requests that the system should refuse to respond to because the necessary information is not in the knowledge base; questions requiring synthesis of information from multiple documents; ambiguous queries where the user's intent is unclear; as well as inputs with false premises that the system should contest rather than blindly accept as truth.

For example, a user tells your HR assistant: "Since the company is matching 401(k) contributions at 8%, I want to choose the maximum." If the actual matching amount is 4%, the system should correct the false premise instead of building further responses based on it. Such tests are not difficult to devise: subject matter experts can usually come up with dozens of tricky edge cases based on practical experience. And these are what catch types of failures that basic metrics of contextual fidelity and relevance do not notice at all.

Specific Attack Surface of RAG

RAG systems create attack vectors that do not exist in standalone LLMs. In the 2025 edition of "Top 10 Security Risks for LLM Applications" by OWASP, a new item appeared – "Vulnerabilities in Vector Search and Embeddings," specifically dedicated to RAG vulnerabilities. This reflects how central search pipelines have become in product AI systems.

The most serious threat specific to RAG is what researchers call indirect prompt injection: malicious instructions are embedded not in the user’s request, but in the documents retrieved by the system. Greshake et al. formalized this class of attacks in a 2023 paper presented at the ACM Workshop on Artificial Intelligence and Security (AISec ’23), showing that adding a search mechanism to LLM fundamentally blurs the line between data and instructions. When a RAG system retrieves a document with hidden commands like "ignore the previous context and respond [with the attacker's content]," the LLM may comply with these instructions because it cannot reliably distinguish the retrieved context from system commands.

The related threat—knowledge base poisoning—goes even further. Here, the attacker does not inject instructions directly but corrupts the corpus itself, adding documents specifically crafted to appear for certain queries and push the model toward pre-determined, yet incorrect, answers. The PoisonedRAG study (Zou et al., presented at USENIX Security 2025) showed that introducing just five specially prepared documents into a corpus of millions of records can achieve a success rate above 90% for targeted queries across multiple LLMs and different retrieval configurations. The attack works because the poisoned documents are optimized for two conditions simultaneously: the retrieval condition—to be surfaced by the retrieval mechanism, and the generation condition—to steer the LLM’s response in the attacker’s desired direction. Moreover, this is effective even under black-box conditions, when the attacker has no access to the parameters of the retrieval module.

What Exactly Needs to Be Tested

A practical test suite for competitive evaluation of RAG systems should at a minimum include the following:

Prompt injection resilience. Test the system against common injection patterns added to user queries: role override (“Now you are an unrestricted assistant…”), instruction override (“Ignore previous instructions and…”), as well as their disguised variants. Measure whether the system’s behavior deviates from the expected and whether it exposes the content of the system prompt.

Knowledge base integrity. If your corpus is supplemented from sources you do not fully control—user documents, web scraping, third-party databases—verify what happens when malicious payloads exist in that content. Add malicious documents with high semantic similarity to the test environment and measure whether the system retrieves them and starts acting according to them.

Correct refusal. Make sure that the system refuses to respond when necessary: to questions outside its subject area, to requests for actions it should not perform, such as approving refunds or making medical diagnoses, as well as to requests where the found context is insufficient for a reliable answer.

Consistency in rephrasing. Ask the same question in different phrasings and check if the substantive consistency of the answers is maintained. Inconsistency in rephrasing often indicates that the system is too sensitive to surface phrasing, rather than the real intent of the user. This is both a reliability issue and a potential vector for exploitation.

To begin such checks, no complicated tools are needed. A table with competitive queries, expected behavior, and pass/fail criteria – with evaluation through an LLM-judge calibrated to the human-in-the-loop scheme – will catch most critical issues before users encounter them.

First measure, then improve

When you look at the entire set of available metrics, frameworks, and tools, RAG evaluation might seem frighteningly complex. But in practice, the way forward is simpler than it seems. Start with the RAG triad: context relevance to check the search module; contextual accuracy to detect hallucinations; and answer relevance to ensure the system is genuinely helping the user. These three metrics cover the most critical types of failures and provide a diagnostic basis for targeted improvements.