Build an evidence checklist for AI assisted changes

An assistant reports “implemented and tested” after running a type check. The feature also changes authorization, database writes, and error recovery, none of which the type check exercised. The problem is not simply that more tests are needed. The completion claim has no clear relationship to the evidence.

An evidence checklist makes that relationship explicit. It describes what must be true, how it was checked, and which boundaries remain unverified. Consider an illustrative profile-image upload change that adds a server-side size limit and clearer client feedback.

Begin with claims rather than commands

List the claims the change makes. Oversized uploads are rejected by the server. Permitted images are stored successfully. Users cannot modify another account's image. Failed storage does not produce a success message. The page preserves a useful recovery path.

Then connect each claim to an evidence method. A server test can exercise the size boundary; an integration test can verify provider storage; a two-account test can verify authorization; a browser check can verify the retry interaction. No single command proves the entire list.

Avoid treating a green command as an abstract badge of quality. “Type check passed” is a precise statement about static compatibility within its configuration. “Upload is secure” is a much broader claim. Good evidence language keeps the scope of the statement matched to the scope of the check.

Record the original failure

Include a reproduction or baseline observation before the correction. In the upload example, a synthetic file above the agreed limit reaches the storage operation. The reproduction should identify the route, account context, input size, and observed status.

Preserve enough detail to repeat it. A phrase such as “large uploads fail” is ambiguous: fail where, with which response, and after which side effects? A controlled example can establish whether the limit applies before storage and whether the client receives a meaningful error.

If the change is a new feature, use acceptance examples as the baseline instead of inventing a prior defect. Evidence should explain the intended behavior honestly. A feature addition and a bug correction have different starting points, even when both need verification.

Choose checks at the relevant boundaries

Static checks catch imports, syntax, and type mismatches. Unit tests exercise local decisions. Integration tests exercise boundaries with storage or providers. End-to-end checks connect the user interaction to the deployed or isolated application. These layers overlap but do not replace one another.

For upload limits, test below, at, and above the agreed boundary using the application's actual interpretation of size. Check missing metadata and malformed content where they affect the contract. A client-only size check does not protect the server against a direct request.

Inspect side effects. Rejection should not leave an orphaned object if the intended rule is validation before upload. Storage failure should not replace the user's current image with a broken reference. The evidence checklist should protect the invariant, not just the response message.

Give each item a truthful status

Use statuses such as passed, failed, not run, and not applicable. Include the command or action, relevant environment, and short result. “Not run: requires isolated object storage credentials” is useful information; silently omitting the integration item is not.

A skipped item is not automatically a blocker. Its importance depends on the changed behavior and remaining risk. A copy-only UI change may not need a storage integration test. A new provider adapter likely does. State why an item matters so reviewers can make a proportionate decision.

Distinguish planned verification from completed verification. An assistant can produce an excellent test plan without running it. The final report should not blend those two categories into “tests covered.” Ask for exact executed checks and their outcomes.

Use AI to find gaps, then verify its checklist

Provide the acceptance examples and the changed paths. Ask the assistant to map each claim to evidence and identify missing boundaries. Require it to avoid proposing broad unrelated suites unless they address a concrete risk.

An illustrative request is: “For this upload change, list the behavioral claims, existing tests that support them, and unverified provider or browser boundaries. Do not mark a check passed unless it ran. Identify a regression case that would fail before the fix.” This makes the assistant useful as a coverage analyst.

Inspect its mapping. It may claim that a mocked provider test verifies actual storage access or overlook a route that bypasses validation. The guide on reviewing plausible code describes how to trace those paths before relying on a summary.

Make evidence easy to review

Keep the checklist near the change description or a concise verification note. Link to the tests and include a short reproduction. Avoid pasting enormous logs unless a specific failure needs them. Reviewers need conclusions supported by inspectable artifacts, not a transcript of every command.

Name material environmental differences. A local filesystem adapter does not prove the cloud provider's permissions. A development build may not reveal serverless request limits. If those differences affect the upload path, include a deployment-specific check.

Do not invent benchmarks or production outcomes. If the change was verified only with synthetic inputs, say so. Evidence from a small controlled test can still be strong for a size predicate while remaining insufficient for throughput claims.

Keep the checklist proportional and durable

A useful checklist is short enough to use repeatedly but specific enough to expose gaps. Start with the change's risks and reuse stable categories only where they help. An exhaustive template that people routinely skip creates less confidence than a compact note they complete honestly.

When a recurring defect appears, update the relevant checklist item or automated check. If uploads repeatedly leave orphaned objects, make cleanup behavior explicit. Do not add vague reminders such as “consider robustness” when a concrete failure mode is known.

Completion becomes reviewable when the code, reproduction, and evidence agree. The checklist does not guarantee perfection. It gives the team a clear account of what was demonstrated, what remains uncertain, and why the proposed change is reasonable to accept.

Include a rollback or recovery observation when the change alters durable state. For the upload example, determine whether a failed replacement keeps the prior image reference and whether partially created objects can be cleaned up. A successful new upload alone does not prove the replacement path is safe. Record the tested failure point and resulting state so reviewers can see what survives interruption. This extends the checklist from response correctness to the user’s existing data without requiring a broad disaster-recovery exercise for every small edit.

Sources

GitHub Copilot best practices advises validation with review and tooling.

Anthropic: Demystifying evals for AI agents discusses evaluation structure and reliable checks.