Review AI generated tests for false confidence

AI-generated tests can look thorough while proving very little. They may import the wrong function, use fixtures that avoid the risky branch, catch errors without failing, or assert only that a mock was called. A long test file and a green run are not enough. Review must ask whether plausible wrong behavior would still satisfy the assertions.

Consider an illustrative API fix that prevents users from changing another account's profile. The assistant generates success-path tests and a denied request test, but the denied case checks only the status code. If the handler mutates data before returning forbidden, the test still passes. This example shows why test review needs behavioral counterexamples, not just syntax inspection.

Start with the behavior claim

Write the invariant in plain language. An unauthorized actor cannot change the target profile and cannot read private target fields. An authorized actor can change permitted fields without changing ownership or role. These claims guide fixture choice and assertions.

Ask the assistant which generated test establishes each claim. Require file and assertion references rather than a summary saying comprehensive coverage. If a claim has no assertion, it is not verified by the suite. Separate build, unit, integration, and end-to-end evidence. A unit test of the permission helper does not establish that the API route supplies trusted identity or invokes the helper before mutation. The intended boundary must be explicit before review can judge whether the test reaches it.

Inspect fixture authority and realism

Check whether the denied actor is genuinely different from the resource owner. Shared fixture helpers may assign administrator privileges or use the same identifier for every account. Verify tenant, role, session, and resource relationships. A fixture with impossible state can also produce misleading results by exercising a branch that never occurs in the application.

Keep data synthetic and minimal, but retain the distinctions the bug requires. For profile access, use two independent accounts and a private field with a recognizable test value. Record which fields are meaningful. Do not copy real user records to obtain realism. The fixture should explain the policy condition clearly enough that a reviewer can predict the expected outcome without reading every helper implementation.

Look for assertions that cannot fail meaningfully

Search for truthy checks on objects that always exist, expectations inside unawaited callbacks, and tests that catch errors and continue. Check whether asynchronous operations are awaited and whether the runner discovers the intended files. A test can execute without reaching its assertions if setup returns early or an optional dependency causes a skip.

Inspect negative assertions carefully. Checking that an error message exists may pass for an unrelated database failure. Assert the expected outcome category and durable state. For the unauthorized profile update, reload the target profile and confirm permitted and forbidden fields remain unchanged. Check response field absence where privacy is part of the invariant. These assertions make a plausible mutate-then-deny implementation fail.

Challenge the suite with wrong implementations

Use a mental counterexample or a controlled mutation experiment. What if the route ignores ownership? What if it updates first and checks permission later? What if it trusts the role in the request body? Ask whether the current assertions would detect each mistake.

Where practical, temporarily introduce a bounded incorrect behavior in an isolated checkout and confirm failure, then restore it safely. Do not damage unrelated work or claim mutation testing was performed if it was only discussed. A practical artifact lists incorrect behavior, expected failing test, and observed result or reasoning limitation. This review often provides more confidence than adding several redundant happy-path cases, because it tests the sensitivity of the suite to the actual risk.

Verify real boundaries are not mocked away

Inspect mocks around authentication, permissions, database access, and provider adapters. A mock permission function returning denied can verify response mapping while bypassing the ownership rule. A fake database can hide query scoping or uniqueness behavior. Label those tests accurately and add targeted boundary evidence where needed.

The right answer is not to remove every mock. Doubles are valuable for controlled failures and fast feedback. The issue is whether the mocked boundary is the one the test claims to verify. See provider doubles for keeping that scope explicit. A reviewable suite can combine narrow tests with a few realistic integration cases without pretending each test proves the whole system.

Check synchronization and cleanup

Replace arbitrary timing assumptions with observable waits or controlled scheduling. A fixed sleep may hide a race locally and fail on CI. Confirm timers, mocks, records, and files are cleaned up so order does not change the result. Use run-specific synthetic identities for parallel cases.

Review failure cleanup too. A barrier or deferred promise should not leave the process hanging when setup fails. A test that depends on another test's data is fragile even if the current order passes. Run the relevant suite in its intended mode and inspect actual discovery and results. If only one test was run manually, do not report the complete suite as verified. Honest execution scope is part of test quality.

Keep the suite focused and readable

Remove generated repetition that adds no distinct behavior. Prefer case names that explain actor, condition, and outcome. Keep setup near the test or in a helper whose meaning is clear. An abstract factory with many default privileges can make policy tests difficult to audit.

Do not assert private implementation details unless they are the relevant contract. Tests tied to helper names and call order often fail during safe refactoring while missing real behavior defects. Choose assertions that survive a correct alternative implementation. Maintain a small evidence map connecting critical invariants to tests. This makes future changes easier to review and helps both people and assistants avoid describing line count as coverage.

Give AI a skeptical review prompt

Supply the bug statement, generated tests, fixture helpers, and relevant implementation. Ask the assistant to propose three plausible wrong implementations that would pass, identify swallowed failures, and mark mocked boundaries. Require concrete evidence for findings. Do not ask it simply whether the tests are good, which invites a broad reassuring answer.

Review suggested repairs and run them in the proper environment. Verify any invented testing APIs against installed versions. The final report should say which tests ran, whether the original defect was shown to fail before the fix, and which infrastructure remains untested. AI can accelerate test review, but confidence comes from meaningful assertions and execution evidence rather than the assistant's approval of its own output.

Sources

Node test runner documentation provides the primary reference for asynchronous tests and mocking behavior.

Playwright best practices support behavior-focused assertions and isolation. The profile counterexamples are illustrative test-review exercises.