How to Evaluate RAG Systems in Production: Metrics and Tests

Why RAG Works in a Demo but Fails in Production

A retrieval-augmented generation (RAG) system can produce convincing answers during a demonstration and still fail when real users depend on it. The problem may begin before the language model generates a response: the correct document was never retrieved, an outdated version ranked higher, or important information was lost when source files were processed.

Even when retrieval works, the model can misinterpret evidence, combine facts from unrelated records or answer confidently when the available information is insufficient. For enterprise engineering teams, production readiness therefore requires more than testing a handful of prompts. It means measuring retrieval quality, answer correctness, groundedness, security and operational performance separately, then evaluating how they work together. According to VentureBeat, connecting a language model to company documents is relatively straightforward compared with managing the data quality, permissions and reliability required for business use.

Why a successful RAG demo is not proof of production readiness

A demonstration usually starts with a small, carefully selected collection of documents and questions that are easy to answer. The developer knows what information exists and can often adjust the prompt or retrieval settings until the output looks right.

Enterprise knowledge bases are less predictable. Policies conflict, spreadsheets contain information that document parsers miss, and important answers may be buried in long PDFs or internal support records. Documents also change, while access permissions differ between employees.

Imagine an employee asking an internal assistant whether a particular service includes a specific feature. The system retrieves a previous product policy instead of the current one. The answer may be grammatically correct, supported by a real document and completely wrong for the employee's question.

A better embedding model will not necessarily solve this problem. The source document may be outdated, incorrectly indexed or missing essential information.
Before changing the model or prompt, teams should identify which part of the pipeline failed.

The four failure points: ingestion, retrieval, context and generation

A RAG application has several connected stages, and each can introduce a different type of error.

Ingestion: The system extracts information from source documents. Incorrectly parsed tables, missing PDF text or lost metadata can make useful information unavailable to search.

Retrieval: The search system selects and ranks passages for a query. Poor chunking, ambiguous terminology or unsuitable search settings can prevent relevant evidence from appearing in the results.

Context construction: The application chooses which retrieved passages to send to the model. Important evidence may be truncated, buried beneath irrelevant material or mixed with conflicting versions.

Generation: The language model uses the supplied context to answer. It may omit important details, misinterpret a passage or introduce claims that the evidence does not support.

A reliable evaluation process needs to isolate these stages. If the correct passage never reaches the model, rewriting the generation prompt may have little effect. If the evidence is correct but the answer is misleading, the investigation should focus on context construction, generation and response validation.

Build a representative evaluation dataset

Before tuning a RAG system, create a collection of realistic questions with verified evidence and expected outcomes. This becomes the baseline against which developers can compare changes to the retriever, model, prompt or document-processing pipeline.

Start with questions from actual users where available, supplemented by subject-matter experts who understand the business domain. Include common requests, ambiguous wording, multi-document questions, questions about recent changes and questions for which the approved knowledge base contains no answer.

For each test case, record the question, the authoritative source documents, relevant passages, expected answer or required facts, and any access restrictions.

Consider a policy assistant answering questions about employee leave. The dataset should include ordinary questions about leave allowances, questions involving exceptions, questions where an old policy conflicts with a new one, and questions that cannot be answered from the approved documents.

Keep a held-out test set that is not repeatedly used to tune the system. Otherwise, developers risk optimising for familiar examples rather than improving performance on new queries.

OpenAI's evaluation best practices recommend task-specific datasets, repeated evaluation, human review and continuous testing. Microsoft also documents separate retrieval and response evaluators in its RAG evaluation guidance.

Measure retrieval quality separately from answer quality

Retrieval metrics reveal whether the system finds the evidence it needs. Answer metrics assess what the model does with that evidence.

These metrics answer different questions. A system may achieve strong retrieval recall but produce incomplete answers. It may also generate a relevant response from incomplete evidence, making it appear useful even though it misses a critical detail.

Frameworks such as Ragas provide metrics for context precision, context recall, faithfulness and response relevance. Microsoft's RAG evaluators likewise distinguish document retrieval from final-answer evaluation. No single score establishes production readiness. The appropriate thresholds depend on the use case, the consequences of an error and the quality requirements agreed upon by the business.

Example of Context Recall:

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextRecall

# Setup LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)

# Create metric
scorer = ContextRecall(llm=llm)

# Evaluate
result = await scorer.ascore(
    user_input="Where is the Eiffel Tower located?",
    retrieved_contexts=["Paris is the capital of France."],
    reference="The Eiffel Tower is located in Paris."
)
print(f"Context Recall Score: {result.value}")

Output:

Context Recall Score: 1.0

Test citations, insufficient evidence and access permissions

A response that includes citations is not automatically trustworthy. The cited passage may be unrelated to the claim, support only part of a sentence or come from a superseded policy. Evaluate whether citations point to the correct source and whether that source supports the specific statement beside them. Where possible, test each material claim against the underlying evidence rather than checking only whether a citation exists.

The system should also be tested on questions that cannot be answered from the available information. A dependable assistant should be able to acknowledge missing evidence, request clarification or direct the user to an appropriate source instead of manufacturing an answer.

Access control requires a separate set of tests. If a user cannot read a confidential document in the original application, the RAG system should not expose its contents through generated answers. Permissions should be enforced before restricted information enters the model's context, not left solely to the model's judgment.

Finally, test data freshness. When a policy is replaced or a document deleted, verify that obsolete information no longer appears in retrieval results and that the system can identify which source version informed a response.

Test latency, cost and performance under load

A system that answers correctly but takes too long or costs too much per request may still be unsuitable for production. Measure end-to-end latency, retrieval time, generation time, token usage, error rates and cost per successful query. Test representative workloads, including concurrent users and longer questions that retrieve several passages.

Additional retrieval stages or reranking may improve relevance but increase latency and operating costs. Increasing the number of retrieved chunks can improve recall in some cases while introducing irrelevant or contradictory context in others.

The objective is to find a configuration that meets the business's quality requirements within its performance and cost constraints. Those targets should be defined for the actual application rather than borrowed uncritically from a general benchmark.

Make evaluation part of every release

RAG quality can change when a team updates the embedding model, modifies chunk sizes, changes the prompt, introduces a new data source or replaces the language model. Every meaningful change should trigger regression tests against the established evaluation dataset.

After deployment, monitor anonymised or appropriately protected query patterns, user feedback, retrieval failures, citation problems, latency and cost. Add newly discovered failure cases to the evaluation dataset so the system becomes more robust as usage changes. Automated evaluators can help process large volumes of outputs, but they should be calibrated against human judgments, particularly for complex or high-impact questions. A model-based judge can make mistakes too.

Production readiness is a continuing engineering responsibility

RAG can give language models access to current, domain-specific information, but that capability depends on the quality and control of the entire pipeline.

For teams, the practical starting point is straightforward: assemble a representative evaluation dataset, test retrieval and generation independently, verify citations and permissions, measure performance under realistic workloads, and rerun those tests whenever the system changes.

The aim is not to prove that a RAG application never fails. It is to understand its failure modes, set acceptable limits, detect regressions and prevent unreliable answers from becoming consequential business decisions.

A successful demo shows what a system can do under favourable conditions. A production evaluation establishes how reliably it performs when the questions, data and operating conditions become less predictable.

Read more: OpenAI AI Agents Weakened Internal Monitoring. What Enterprise Teams Should Learn
Read more: Part 2: Andrew Ng’s AI Skills Map: 6 Skills African Should Learn to Build Real AI Products
Read more: How to Build an App With AI: A Complete Guide for Beginners