Would your AI-generated tests catch the wrong result?

Start with a result the business can check

Use a hypothetical delivery rule: standard shipping costs £4.99 below a £50 order subtotal and is free at £50 or more. This example excludes other charges, promotions, and invalid inputs.

Three examples make the threshold unambiguous:

• £49.99 subtotal: £4.99 shipping
• £50.00 subtotal: free shipping
• £50.01 subtotal: free shipping

Agree those outcomes before asking AI to generate tests. A test built only from the current implementation may preserve its misunderstanding of “at least £50.”

Suppose the implementation mistakenly gives free shipping only above £50. This plausible test repeats that rule when calculating its expected result:

// Inside the same Node test setup used below:
test('shipping at the threshold', () => {
  const subtotalPence = 5000;
  const expected = subtotalPence > 5000 ? 0 : 499;
  assert.equal(shippingPence(subtotalPence), expected);
});

Both implementation and test charge £4.99 at exactly £50, so the test passes. Their agreement hides the shared mistake. Microsoft recommends keeping logic out of unit tests to avoid defects in the checks themselves. Unit-testing best practices

Use explicit expected values instead. With Node.js 24, save the following as two files and run node --test shipping.test.mjs. Amounts are integer pence.

// shipping.mjs
export function shippingPence(subtotalPence) {
  return subtotalPence >= 5000 ? 0 : 499;
}

// shipping.test.mjs
import test from 'node:test';
import assert from 'node:assert/strict';
import { shippingPence } from './shipping.mjs';

test('£49.99 still pays shipping', () => {
  assert.equal(shippingPence(4999), 499);
});
test('£50 qualifies for free shipping', () => {
  assert.equal(shippingPence(5000), 0);
});
test('£50.01 still qualifies', () => {
  assert.equal(shippingPence(5001), 0);
});

These use Node’s built-in test runner. Node.js test documentation

A pass is the starting point. Can the tests detect a fault?

In a temporary local copy, change >= to >. Leave the tests unchanged. The £50 case should now fail because the implementation charges a customer who qualifies for free delivery.

Restore the code, then try two more small faults: always return zero, and always return 499. The below-threshold case should catch the first; the qualifying cases should catch the second. Restore the original implementation after every experiment.

This is a small manual form of mutation testing: deliberately alter behavior to see whether the tests notice. A surviving change deserves investigation, although some changes are behaviorally equivalent and do not represent a missing test. Stryker mutation-testing states

In an isolated Node.js run, the explicit checks passed for the correct function and detected all three faulty variants. The duplicated expected-value calculation passed the faulty > implementation; the independently specified £50 case failed it. These results establish sensitivity to those faults, not complete pricing coverage.

Check what the mock has removed

Suppose a browser test intercepts the shipping API, returns a fee of zero, and checks that the checkout displays “Free.” That can test the display. It cannot establish whether the real API calculated the correct charge.

Playwright’s documentation makes this boundary explicit: fulfilling an intercepted request with mock data can mean the API is never called. Playwright API mocking

Challenge that display test too. In a temporary copy, change the response fixture from zero to 499 while keeping the original “Free” assertion. A renderer using the supplied value should make the assertion fail. If it stays green, investigate whether the page hard-codes the label, the mock matched the wrong request, or the assertion inspected the wrong element.

Then restore the original test and add a proper paid-shipping case expecting “£4.99.” An offline rendering model demonstrated why: a hard-coded “Free” renderer passed the free-only case but failed the independently specified paid case. This was a pure-function check, not a browser run.

Keep a separate check of the real shipping implementation. Name each test according to its actual boundary: calculation, mocked display, or integration with the service.

Start with a bounded repository review

Select a sample of business-critical rules with the process owner and technical lead. Prioritize changed calculations, important exceptions, and the proposed release.

Trace expected values to their business source. Look for shared wrong assumptions, important behavior removed by mocks, and weakened assertions. Apply selected local faults to resolve specific doubts; the entire suite does not need manual mutation.

Deliver prioritized findings: affected rules, test gaps, verified examples, and recommended corrections. Name unreviewed areas so the team can decide what to fix before release and what further review to fund.

Coverage can help locate unexecuted code, but cannot establish whether every executed result was checked correctly. Google’s coverage guidance

The team should leave able to explain what would make important tests fail, with evidence supporting the release decision.