AI-generated unit tests: how do you know they check anything?

Agents love to create a thousand unit tests. All green. Then review smashes the whole thing.

The agent already implemented a trash solution, and only then wrote generalized unit tests for it. Those tests describe the code it just wrote. So they all pass, bugs included.

When an agent writes both the code and its tests, don’t count the green checks. Look at where each expected value comes from. If it comes from the agent’s code, the test can repeat the agent’s mistake instead of catching it. If it comes from the ticket, the spec or a real example, and the test runs the real code and failed because the behavior was missing, it checks something.

Two patterns that pass with the bug

Some tests mock the exact function they’re supposed to test. Others calculate the expected value with the same formula as the code. Both can miss the bug: one replaces the implementation, the other repeats its mistake in the expected value.

Here are both, in TypeScript with Vitest. Suppose an agent writes this for a discount ticket:

// pricing.ts. The ticket: orders of 100 or more get 10% off.
export function applyDiscount(total: number): number {
	return total > 100 ? total * 0.9 : total // bug: should be >=
}

A test that mocks the function it’s supposed to test:

// mocked.test.ts
import { expect, it, vi } from 'vitest'
import { applyDiscount } from './pricing'

vi.mock('./pricing', () => ({ applyDiscount: () => 90 }))

it('gives 10% off at 100', () => {
	expect(applyDiscount(100)).toBe(90) // checks the mock, not the code
})

A test that calculates the expected value with the code’s formula:

// same-formula.test.ts
import { expect, it } from 'vitest'
import { applyDiscount } from './pricing'

it('applies the discount', () => {
	const total = 100
	// the code's own formula, bug included
	const expected = total > 100 ? total * 0.9 : total
	expect(applyDiscount(total)).toBe(expected)
})

Both pass, and the code still charges 100 for an order of exactly 100. With the mock, the real applyDiscount never runs. Mocking a dependency, like a payment service, is a different and legitimate use (Software Engineering at Google). Copying the formula copies the bug. Google’s testing blog shows the same trap in Don’t Put Logic in Tests.

Researchers have measured a version of this. In a July 2026 study, 11 models in 13 model and mode setups wrote tests for 318 Java methods behind 233 real bugs. Averaged across those setups, 3.84% of the tests written from the buggy code passed on the bug and failed on the fix. Written from the fixed code, 0.46% did (Zhao, Zhou and Cohen).

More tests don’t fix it

In a study first posted in February 2026 on 500 SWE-bench Verified tasks, pushing agents to write more tests barely changed how many they solved (Chen et al.). GPT-5.2 solved 359 tasks either way, and Gemini 3 Pro Preview went from 371 to 366. Telling Kimi K2 Thinking and DeepSeek v3.2 Reasoner not to write tests moved them from 317 to 304 and from 300 to 291. None of these changes was statistically significant.

The cost depended on the model: GPT-5.2 made more API calls and used more tokens without solving more, and the two models told to skip tests used fewer (results). It’s one benchmark and one agent scaffold, mini-SWE-agent, and the study didn’t test TDD or the workflow below.

Red and green slices, then review

In my experience, TDD with small red and green slices, and reviewers after, works much better:

1. Define what done means, in a way that can fail.
2. Take the expected values from something the agent didn’t write: the ticket, the spec, a real example.
3. Write one failing test (red). It has to fail because the behavior is missing.
4. Write only enough code to pass it (green).
5. Next slice. Reviewers come in after green.

For the discount, the test takes its expected value straight from the ticket:

// spec.test.ts
import { expect, it } from 'vitest'
import { applyDiscount } from './pricing'

// From the ticket: an order of exactly 100 pays 90.
it('gives 10% off an order of exactly 100', () => {
	expect(applyDiscount(100)).toBe(90)
})

Run against the agent’s code, it fails red with expected 100 to be 90. Change > to >= and it goes green. The mock test still passes, because it never ran the code. The formula test now fails: it was guarding the bug. Checked with Vitest 4.1.12: save the four blocks under their file names and run npx vitest run.

Matt Pocock’s /tdd skill is his version of this workflow. One slice at a time, and writing all the tests up front is an anti-pattern there. Refactoring belongs to review. That part is a choice: Kent Beck’s Canon TDD lets you refactor inside each cycle, after green.

Lauren Tan (poteto), the author of pstack, builds step 1 into it. Its /figure-it-out skill asks the agent to state what done means, in a way that can fail, before it starts the run.

TDD and /figure-it-out can work magic when you understand what you are doing.

What each check covers

Even good unit tests are one row of this chart. Here’s what every row covers, so you can see what your software is and isn’t tested for. The When column is my rough rhythm, not a rule.

Check	When	What it covers	Source
Unit	Every edit	A small unit of behavior, run fast. The unit can be one function or a few that work together.	Fowler
Mocks	In unit tests and narrow integration tests	Stand-ins for real collaborators, like a payment service. A mock also checks how it was called.	Fowler
Smoke	Every build	Does it start, and do the critical basics work, before deeper testing?	ISTQB
Integration	Every PR	Do separately built parts work together once connected?	Fowler
Regression	Every PR	Did the change break something that used to work? Unit, integration and end-to-end tests can all do this job.	ISTQB
Review	After green	Someone other than the author reads the change: a person or a review model.	Google, GitHub
End-to-end	Before release	A whole user flow, start to finish, in production-like conditions.	ISTQB
Security	Throughout development	Whether the security controls hold. “Can someone get what isn’t theirs?” is one question of many.	OWASP, framework
Mutation	Rarely	Change the code on purpose and see whether any test fails. A surviving change isn’t always a real bug.	Stryker
Manual	By a human	A person uses the product and checks that it behaves as expected.	Atlassian

A shorter version of this first appeared on LinkedIn on 2026-10-22.