Citation-Grounded RAG Evaluation: Measure Whether Answers Earn Their Sources

A RAG answer can display five citations and still be unsupported. The documents may be relevant to the topic but not to the specific claims. A source might support half a sentence, contradict a number, or appear too far from the statement it supposedly verifies.

Citation-grounded RAG evaluation separates those failure modes. The objective is not to maximize citation count. It is to measure whether important claims are supported by the retrieved evidence and whether the answer represents that evidence faithfully.

Split Retrieval From Generation

Start by evaluating retrieval independently. For each question, define the evidence needed to answer it and check whether the retrieved set contains that evidence. Measure recall at a fixed context budget, not merely whether one top result looks relevant.

Also inspect distractors. Near-duplicate pages, obsolete policies, and keyword matches can crowd out authoritative passages. Record document version, publication date, access scope, and chunk boundaries so failures can be reproduced.

If the necessary evidence never enters context, generation metrics cannot diagnose the root cause.

Break Answers Into Atomic Claims

Evaluate claims small enough to verify: one entity, relationship, quantity, or condition at a time. “The plan costs $20 and includes unlimited exports” contains at least two claims that may require different passages.

Mark each claim as supported, contradicted, not found, or not requiring external evidence. Then weight by importance. An unsupported transition sentence is not equivalent to an invented eligibility rule.

Automated claim extraction helps at scale, but review its segmentation on complex sentences, tables, and qualifications.

Test Citation Entailment and Placement

For every citation, ask whether the cited passage actually entails the nearby claim. Topic similarity is insufficient. Preserve enough surrounding text to interpret pronouns, exceptions, and table headers.

Then test citation completeness: what fraction of evidence-requiring claims have at least one supporting source? Precision and completeness should be reported together. A system that cites only its safest sentence can achieve high precision while leaving the important claims naked.

Citation placement matters for users. Bind references to the smallest reasonable claim span rather than placing a pile of sources at the end of a long paragraph.

Include Abstention and Conflict Cases

Your evaluation set should contain missing evidence, conflicting sources, stale documents, access-restricted documents, and questions whose premise is wrong. The desired behavior may be to abstain, explain uncertainty, or present the conflict.

Reward calibrated refusal when evidence is absent. Otherwise the benchmark teaches the model to produce an answer regardless of support.

Build a Regression Dashboard

Track retrieval recall, claim support, contradiction rate, citation precision, citation completeness, abstention quality, latency, and context cost. Slice by corpus, question type, document age, language, and answer length.

Keep a human-labeled anchor set and rerun it whenever the embedding model, chunking, reranker, generator, prompt, or corpus changes. Use model judges as scalable reviewers, not ground truth; monitor their agreement with humans.

A trustworthy RAG system does more than retrieve related pages. It exposes a short, auditable path from each consequential claim back to evidence that really supports it.