Start with the job the application must do
A generative AI application can sound convincing while failing at its actual task. A support assistant might write a polite reply but omit the customer’s question. A document summarizer might produce fluent prose while losing a crucial qualification. An evaluation set is a collection of inputs and review criteria that helps you notice failures like these before treating a change as an improvement.
Define the application’s intended job in observable terms. Who provides the input, what information may the system use, and what should its output enable someone to do? Record important constraints, such as preserving uncertainty, using a required format, or declining requests outside the application’s scope. These decisions give reviewers a shared basis for judging results instead of asking whether an answer merely feels good.
Choose cases that reflect real use and meaningful difficulty
Begin with examples of the requests the application is expected to handle. Include straightforward cases, but avoid building a set made entirely of clean, well-written prompts. Real inputs may be brief, ambiguous, repetitive, or missing context. If the application works with documents, vary their structure and include cases where the relevant detail is easy to overlook.
Then add cases that probe likely failure points. For a summarizer, these might include a document with conflicting statements, a statement qualified by an exception, or a request for a fact the document does not contain. For a drafting assistant, include instructions that compete for space or require a particular audience and tone. You do not need every imaginable edge case; choose examples that would teach you something useful if the system failed. Keep a separate set of newly discovered failures so you can decide which deserve a place in ongoing testing.
Write criteria before judging outputs
A reference answer can be helpful when there is a narrow correct result, such as extracting a named field. Open-ended tasks need a different approach: specify what the response must include, what it must not invent, and which variations are acceptable. A summary, for example, may pass if it captures the main decision and its conditions, even if its wording differs from a sample summary.
Use a small set of clear review dimensions rather than one vague quality score. You might assess factual consistency with the supplied material, completeness, instruction-following, and usefulness to the intended reader. Define what a serious failure looks like for each dimension. If two reviewers repeatedly disagree, inspect the criteria and the input: the task may be underspecified. Record a short reason alongside each judgment so the team can distinguish a genuine model problem from an unclear test.
Compare changes without losing the context
Run the same evaluation cases when comparing prompts, models, retrieval settings, or other application changes. Keep the input material and review criteria consistent, and note which version produced each output. Otherwise, a difference in results may come from changed test conditions rather than the change you intended to assess. For subjective tasks, reviewing outputs without showing which version produced them can also help reduce expectations about the newer version.
Look beyond an overall pass count. A change that improves routine answers but introduces failures on sensitive or difficult cases may not be worth adopting. Group results by task type and failure mode, and inspect representative examples. Automated checks can reliably catch some issues, such as missing required fields or invalid output structure. They are less suited to deciding whether nuanced prose faithfully represents a source, where human review may still be necessary.
Maintain the set as the application evolves
An evaluation set is not a one-time exam. As users encounter new tasks or the application gains features, check whether its cases still represent the work it is meant to do. Add examples of important failures after clarifying what the expected behavior should have been. Remove or revise cases whose instructions no longer match the product, but keep a record of those changes so comparisons remain understandable.
Avoid repeatedly tuning an application only to pass a small, familiar set of examples. Doing so can produce reassuring test results without improving broader performance. Keep some cases aside for less frequent checks, and periodically review fresh inputs under the same criteria. The aim is not to prove that the AI always works; it is to make changes easier to assess, expose recurring weaknesses, and give people a clearer basis for deciding when an output needs closer review.
