Test Retries Hide Real Bugs: When a Rerun Helps and When It Lies

Post

Your build is green. One of your tests failed five minutes ago. Both of those things are true, because the test passed on a retry. Here is the short answer. A retry never fixes anything. Keep retries on in CI, but treat a retried pass as a bug report. It opens a ticket. It never closes one.

Last Saturday I posted about rerunning failed tests in isolation. Two readers pushed back, and both were more right than my post.

The first said flakiness is a signal. Turning the alarm off does not put out the fire.

The second was more specific. A team reruns the failed test alone. It passes. Everyone moves on. And the real cause, a race condition, ships to production. A race condition means two things run at the same time and collide.

This post is the concession, and the setup I now recommend.

What the rerun actually changed

Look at what happens when a test fails in CI and someone reruns it alone.

The code did not change. The data did not change. One thing changed: the test ran without the other tests around it.

So the rerun did not prove the test is fine. It proved the test is fine when nothing else is running. That is a different sentence. Your users do not visit your app one at a time.

The race that a rerun hides

Here is the simplest version of the trap:

test('admin can rename a user', async () => {
  await renameUser('user-42', 'New Name');
  await expect(profileName).toHaveText('New Name');
});

test('report shows user names', async () => {
  const report = await openReport();
  await expect(report.row('user-42')).toContainText('New Name');
});

Both tests touch the same record. Run together, they sometimes collide, and one fails. Run alone, each passes every time.

The failing run was the only honest one. It was telling you the app has a timing bug. The rerun in isolation deleted the evidence.

Playwright already tells you the truth

Here is the part most teams never look at. When a Playwright test fails and then passes on a retry, the report does not say "passed."

It says flaky.

  3 passed
  1 flaky

The tool is honest. The habit is not. Most dashboards only show green or red, so a flaky pass reads as a pass, and the count nobody reads keeps growing.

The setup that keeps retries useful

Retries have one honest job: telling infrastructure noise apart from real signal. A container that started slowly is noise. A race condition is signal. You cannot tell them apart without evidence, so collect it:

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: { trace: 'on-first-retry' },
});

Two lines, two jobs. The retry detects. The trace records everything about the failing run, so you can diagnose it later instead of shrugging.

Locally, retries stay at zero. On your own machine you want the failure loud and immediate.

The rule

A retried pass opens a ticket. It never closes one.

Once a week, read the flaky list. Every entry is one of two things. Infrastructure noise you should fix in the pipeline. Or a real timing bug in the app that your tests found first. Both are work. Neither is "passed."

My Saturday post said rerunning a flaky test in isolation makes things worse. These two readers explained the mechanism better than I did. The rerun does not just waste time. It manufactures false confidence and deletes the only evidence you had.

The green build is not the goal. The true build is.

---

Anton Gulin is the AI QA Architect, the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.