Build a small evaluation set before changing your AI prompt

A prompt change produces a better answer to the example on your screen. You deploy it, then discover that shorter questions now receive invented details and unsupported requests no longer trigger a fallback. One attractive response did not represent the feature's real workload.

A small evaluation set is a practical defense against this tunnel vision. It does not need to begin as a large benchmark. It needs to capture the behaviors the product depends on and provide a repeatable comparison. This article uses an illustrative document-summary feature for internal support notes.

Define the feature's contract

Specify what the summary should include and exclude. It might report the customer's problem, attempted steps, current status, and unanswered question. It should preserve uncertainty and omit information not present in the note. These requirements are more useful than “produce a helpful summary.”

Decide what counts as a severe failure. Inventing a successful resolution may be more consequential than choosing an awkward heading. Distinguish factual errors, missing required facts, format failures, and stylistic preferences. The evaluation should not treat every discrepancy as equally important.

Include the fallback contract. An empty note, corrupted input, or unsupported request should produce an explicit outcome. If the summary is for human review, the product can allow uncertainty labels. If another system consumes it automatically, shape and semantics need stricter validation.

Assemble representative cases

Start with a compact set spanning the real input dimensions: short notes, long notes, contradictory updates, unresolved issues, irrelevant text, and multiple customers mentioned in one document. Use synthetic notes or approved sanitized material rather than casually copying private records.

Give each case an identifier, input, expected facts, forbidden claims, and any acceptable alternatives. A summary can use different wording while preserving meaning. Exact string comparison would reject good paraphrases, so store semantic expectations rather than one mandatory paragraph unless wording is part of the contract.

Add a case for every known failure. If the old prompt confuses “customer will try restart” with “restart resolved issue,” include that distinction explicitly. This makes the set accumulate useful knowledge instead of becoming a random collection of easy examples.

Keep evaluation separate from prompt examples

A prompt may contain demonstrations, but an evaluation set should include cases the prompt has not simply memorized. Otherwise you measure similarity to your teaching examples rather than general behavior.

Separate development cases from a small holdout set if the workload supports it. Use development cases to diagnose failures and retain holdout cases for a less biased comparison. For a tiny early feature, even labeling which cases influenced a prompt edit is better than pretending every result is independent.

Avoid tuning repeatedly to one stubborn example without checking the rest. A prompt instruction that fixes one contradiction may cause the model to overstate uncertainty everywhere. Evaluations are valuable because they reveal these interactions across cases.

Choose checks that match the outputs

Use deterministic checks for required keys, length limits, and allowed labels. Use factual assertions for facts that must appear or must not appear. For subjective quality, use a written rubric and human review of a manageable sample.

A model-based grader can assist with semantic judgments, but it needs its own validation. Check whether it catches deliberately bad summaries and accepts reasonable paraphrases. Its verdict is not independent truth simply because it comes from another model.

Record grader disagreements. If humans cannot consistently decide whether a summary is acceptable, the contract may be unclear. Resolve the policy question instead of adjusting the prompt until one grader gives a high score. Evaluation quality depends on stable expectations.

Run comparable experiments

Keep the same inputs, configuration, and scoring approach for the old and candidate prompts. Record model identifier, prompt version, and relevant generation settings. If you change both model and prompt, attribute the result to the combination rather than claiming the prompt alone improved behavior.

Run repeated trials where output variability matters. A single pass can miss intermittent failures. Choose the repetition count proportionate to the risk and budget, and report the actual trial count. Do not infer a production failure rate from a tiny development sample.

Inspect case-level outcomes, not just an aggregate score. A candidate can improve many easy cases while introducing one severe factual error. The decision should reflect the product's risk priorities. Averages are convenient summaries, not substitutes for examining material regressions.

Use AI to expand and diagnose the set

Give an assistant the feature contract and existing failures. Ask for candidate edge cases that exercise specific distinctions, such as planned versus completed actions or conflicting dates. Review each proposed case before adding it.

An illustrative request is: “Generate synthetic support notes that challenge preservation of unresolved status. For each, list required facts and claims a summary must not invent. Do not reuse the examples already in the prompt.” This is a useful creative task with an inspectable output.

Ask the assistant to categorize failures from the evaluation run. It may suggest that several cases fail because the prompt treats the last sentence as authoritative. Verify that explanation against outputs before changing the prompt. The guide on structured output covers the next boundary when summaries become machine-readable records.

Decide what the evidence supports

A small set can show that a change improves known cases without introducing observed regressions. It cannot prove universal quality or establish a precise production accuracy rate. State its scope and preserve cases that reflect new failures after release.

Choose release criteria before viewing the candidate's results where practical. For example, no severe invented-resolution errors and no new format failures may be necessary, while minor wording differences remain acceptable. This reduces the temptation to redefine success around the current output.

The evaluation set becomes a maintained product artifact. It records what the feature is supposed to do, supports future prompt and model changes, and makes discussion concrete. Instead of debating whether an answer feels better, the team can compare observable behavior across cases that matter.

Keep case identifiers stable when inputs are revised. If a note changes to clarify an expectation, record that revision rather than silently replacing the old case while comparing historical scores. Otherwise apparent improvement can come from an easier evaluation rather than a better prompt. Store the rubric and fixtures with the feature, review changes like code, and retain a short explanation for significant additions. This makes evaluation results interpretable months later when model configurations and product requirements have both changed.

Sources

Anthropic: Demystifying evals for AI agents discusses evaluation components and grader choices.

Anthropic: Building effective agents discusses programmatic checks within workflows.