Provenance Tracking for Trustworthy RAG Answers

Many retrieval-augmented generation systems add citations after the answer is written. The result looks trustworthy, but the links may point to documents that are merely related—not to the passages that support each claim.

Real provenance is a chain of custody. It records where content came from, how it was transformed, which exact chunks were retrieved, what the model saw, and how final claims map back to evidence.

Preserve Identity at Ingestion

Assign stable IDs at every layer: source, version, document, and chunk. Keep the original URI, content hash, retrieval timestamp, access scope, and transformation history.

type EvidenceChunk = {
  chunkId: string;
  documentId: string;
  sourceVersion: string;
  sourceUri: string;
  contentHash: string;
  charStart: number;
  charEnd: number;
  text: string;
  acl: string[];
};

Offsets make it possible to reconstruct the chunk in the source version. A content hash detects silent changes. Version identity matters because a policy page viewed today may not say what it said when the answer was generated.

Transformations should be explicit. If a PDF was OCR’d, a table flattened, or boilerplate removed, store that operation. Otherwise reviewers cannot tell whether an apparent source error originated in the document, parser, or model.

Carry Evidence Through Retrieval

Never pass naked text strings to the generator. Wrap every chunk with its immutable evidence ID and metadata. Record the query, filters, scores, reranking result, and final context order in the trace.

Access control belongs in this layer. A citation must not leak the existence or title of a document the user cannot access. Filter before retrieval when possible and enforce permissions again when rendering source links.

Deduplicate overlapping chunks so repeated text does not appear like independent confirmation. When multiple sources repeat the same claim, retain their separate identities but avoid inflating confidence simply because syndicated copies rank highly.

The prompt should instruct the model to cite evidence IDs, not invent URLs. Your application resolves those IDs into approved labels and links after generation. This removes a major source of fabricated citations.

Map Claims, Not Paragraphs

Ask the model for a structured answer containing atomic claims and supporting evidence IDs. A paragraph may combine several facts, each with different support.

{
  "claims": [
    {
      "text": "The policy took effect in July.",
      "evidence": ["chunk_7f2"],
      "support": "direct"
    }
  ]
}

Validate that every evidence ID was present in the supplied context. Then run an entailment or evidence check: does the cited passage actually support the claim, and is the wording appropriately qualified? Unsupported claims can be removed, regenerated, or clearly labeled as inference.

Some answers should say “the available sources do not establish this.” Provenance makes abstention a product feature rather than a model failure. It also distinguishes source disagreement from retrieval failure.

Make the Audit Useful

Users need concise citations; operators need the full trace. The visible interface can show source title, relevant passage, publication date, and a link. The internal record should also retain model release, prompt version, retrieval configuration, evidence set, and validation results.

Measure citation precision, claim coverage, broken-link rate, stale-source rate, and permission violations. Sample answers for human review, especially in high-impact workflows. A high retrieval score does not guarantee that a passage supports the generated sentence.

Plan for deletion and retention. If a source must be removed, provenance IDs let you find affected answers and caches. If sensitive content should not persist, store hashes and references according to policy rather than retaining every prompt indefinitely.

Trustworthy RAG is not created by placing footnotes at the end. It comes from maintaining evidence identity through ingestion, retrieval, generation, validation, and display. When that chain is intact, a reader can move from a claim to the exact source version that justified it—and an engineer can explain what went wrong when the chain breaks.