How to evaluate a RAG system: metrics, test sets and RAGAS

Why "it looks good" is not enough

Most RAG prototypes are judged by trying a few questions and reading the answers. That works for a demo, not for a system people rely on. Small changes (a new chunk size, a different embedding model, an updated prompt) can quietly make answers worse, and you will not notice until a user does.

Evaluation turns "it looks good" into numbers you can compare after every change.

Evaluate the two halves separately

A RAG system has two jobs, and they fail in different ways:

1. Retrieval: did we find the right passages?
2. Generation: did the model answer correctly from those passages?

If you only score the final answer, you cannot tell which half to fix.

Step 1: build a test set

Start with 30–50 real questions, written the way users actually ask them. For each, record:

• the expected answer (a short reference answer), and
• the source that contains it (document and section).

Include a few unanswerable questions too, where the correct behaviour is "I don't know". Grow the set every time a user reports a bad answer; that is the cheapest high-value data you will ever collect.

Step 2: retrieval metrics

For each test question, compare the retrieved chunks with the expected source:

Metric	Question it answers
Hit rate @k	Is the right source anywhere in the top k results?
MRR (mean reciprocal rank)	How high up does the right source appear?
Context precision	Of what we retrieved, how much was actually relevant?
Context recall	Did we retrieve everything needed to answer?

Hit rate and MRR need only your expected sources and simple code. If hit rate @5 is low, no prompt will save you: fix chunking, add keyword search, or rewrite queries first.

def hit_rate(results, k=5):
    hits = sum(1 for r in results if r["expected_source"] in r["retrieved_sources"][:k])
    return hits / len(results)

Step 3: generation metrics

Metric	Question it answers
Faithfulness (groundedness)	Is every claim in the answer supported by the retrieved context?
Answer relevance	Does the answer actually address the question?
Correctness	Does it match the expected answer?
Refusal accuracy	Does it say "I don't know" for unanswerable questions?

These are usually scored by a strong model acting as a judge, with a clear rubric, or by people for a smaller sample. Spot-check the judge against human ratings before trusting it.

Using RAGAS

RAGAS is a popular open-source library that implements metrics such as faithfulness, answer relevancy, context precision and context recall. You give it, per question, the question, the generated answer, the retrieved contexts and (for some metrics) a reference answer, and it returns scores.

Treat the scores as relative: they are most useful for comparing version A with version B of your pipeline on the same test set, not as absolute truth.

Step 4: make it a habit

• Run the full test set after every change to chunking, embeddings, retrieval settings, prompts or models.
• Keep a simple table of runs: date, change, hit rate, faithfulness, correctness, refusal accuracy, cost, latency.
• Set minimum bars before shipping (for example: hit rate @5 ≥ 0.9, faithfulness ≥ 0.85, all unanswerable questions refused).
• Log production questions with the retrieved sources and the answer, and review the low-confidence ones weekly. Add the bad ones to the test set.

A typical improvement loop

1. Baseline: hit rate @5 = 0.72, faithfulness = 0.80.
2. Smaller, heading-aware chunks → hit rate 0.84.
3. Add keyword (hybrid) search → hit rate 0.91.
4. Stricter "answer only from context, cite sources" prompt → faithfulness 0.90.
5. Similarity threshold → unanswerable questions now refused.

(Illustrative numbers: your results will differ, which is exactly why you measure.)

Checklist

• 30–50 real questions with expected answers and sources
• A few unanswerable questions
• Retrieval scored separately (hit rate, MRR)
• Faithfulness and correctness scored, judge spot-checked
• Results tracked per change, with minimum bars
• Production questions logged and fed back into the test set