Version Your LLM Evaluation Dataset Before the Score Becomes Fiction

Your team improves the prompt and the evaluation score rises from 78 to 86. Then someone discovers that twelve difficult examples disappeared during a spreadsheet cleanup, the grader prompt changed last week, and several labels were silently corrected after the run.

The model may be better. The number cannot prove it.

An evaluation dataset is a software artifact. It needs identity, history, review, and a reproducible connection to every result reported from it.

Give Every Evaluation Run a Complete Identity

A dataset version should identify the exact examples, inputs, reference answers or rubrics, metadata, and exclusions used in a run. Store a content hash or immutable artifact ID rather than relying only on a friendly name such as support_eval_final.

Version the grader configuration too. That includes model, grader prompt, temperature or sampling controls, score schema, parsing logic, and any ensemble or human-review rules. A score produced by a new judge is not directly comparable merely because the test examples stayed fixed.

Record the application side: model snapshot, system prompt, tool definitions, retrieval configuration, policy version, and code commit. The run manifest should let another engineer reconstruct what generated both the answer and the score.

Use one machine-readable manifest as the root:

• dataset artifact ID and hash
• split and slice definitions
• application configuration ID
• grader configuration ID
• execution timestamp and environment
• code commit and dependency lockfile
• per-example outputs, traces, and scores

Separate Dataset Repair From Product Improvement

Evaluation examples are not sacred. Labels can be wrong, customer policy can change, and ambiguous cases should be clarified. The mistake is editing history in place.

When an example changes, create a new dataset version and record the reason. Classify the change: label correction, policy update, new edge case, duplicate removal, privacy deletion, or formatting repair. Then rerun the baseline and candidate against the new version.

This distinction prevents a common illusion. If a candidate performs better only because the dataset was made easier, rerunning the baseline exposes it. If both improve after a label correction, the change belongs to measurement rather than the product.

Keep protected holdouts away from routine prompt tuning. Once engineers repeatedly inspect failures, the set becomes a development set. That is useful, but it no longer estimates generalization in the same way.

Track Slices, Not Just One Average

A global pass rate can rise while critical behavior degrades. Tag examples by language, customer tier, policy topic, tool path, input length, risk level, and known failure mode. Store slice definitions in code so they cannot drift between dashboards.

Set minimum sample sizes and show uncertainty. A jump from 50% to 100% on two examples is a lead for investigation, not a release decision. For safety-critical slices, report counts and individual failures alongside percentages.

Preserve retired examples in history unless privacy or legal requirements demand deletion. Mark why they stopped participating. A removed adversarial case is itself useful evidence about what the team once considered important.

Make Reproduction a Release Gate

Before accepting an evaluation result, rebuild the run from its manifest in a clean environment. Verify that the dataset hash matches, required artifacts exist, and scoring produces the same result within a documented tolerance.

Then compare the candidate and baseline on the same dataset with paired analysis. Investigate examples that flip in either direction. A release note should link to the immutable run rather than paste a number into chat.

Dataset versioning will not make a bad rubric good. It makes changes visible enough to argue about honestly. That is the minimum standard for using evaluation scores as engineering evidence instead of decorative telemetry.