Your CI Retries are Hiding Real Bugs

If you start a Playwright project with npm init playwright@latest, the config it writes retries failing tests twice on CI. That’s a sensible default for a browser suite. Now say a product bug makes one test fail one run in five. With two retries, the test fails all three attempts only 0.2 × 0.2 × 0.2 = 0.8% of the time, so the build stays green in 99.2% of runs.

I planted such a bug in a small service and ran its test suite 41,200 times. Retries hid the bug about as often as the arithmetic predicts, while the test reports flagged the bug’s test as flaky in about one green build in five. Below are the numbers and the quarantine policy I’d build on them: Who owns a quarantined test, how long it may stay and how it gets out.

Test Tools Flag a Pass on Retry, But the Exit Code Doesn’t

The tools I checked do record a pass on retry. Playwright labels the test ‘flaky’, pytest-rerunfailures prints an R for each rerun and Maven Surefire counts it under ‘flakes’. However, the exit code is still 0, and CI passes or fails the job on the exit code. Strict settings that fail the run on flaky passes exist, such as Playwright’s –fail-on-flaky-tests and pytest-rerunfailures’ –fail-on-flaky, but they’re off by default.

Some of those flaky passes are product bugs. Google’s John Micco wrote in 2016 that developers sometimes “dismiss a failing result as flaky only to later realize that it was a legitimate failure caused by the code.” Tests with a flaky record also catch real faults: A study of 10,000 Chromium CI builds (ASE 2024) found that tests that had flaked at some point revealed more than a third of all regression faults.

The Planted Bug: One Order in Five Goes to a Shard That’s Gone

The service under test is a toy order store with five shards. It picks a shard from the last two hex digits of the order ID:

from shop import settings

SHARDS = [“orders-0”, “orders-1”, “orders-2”, “orders-3”, “orders-4”]

def shard_for(order_id: str) -> str:
    return SHARDS[int(order_id[-2:], 16) % settings.SHARD_COUNT]

I built a 24-commit history around it. Commit 18 retires orders-4 by deleting it from the list—however, settings.SHARD_COUNT is still 5, so an ID whose last two hex digits divide by 5 with remainder 4 now raises IndexError. That’s 51 of the 256 possible values, or 19.9% of orders.

The suite’s routing test uses fixed IDs and keeps passing. A roundtrip test catches the bug: It stores an order with a random uuid.uuid4().hex ID and reads it back, so it fails whenever the ID lands on the missing shard.

For background noise, two other tests flake for ordinary reasons: A payment test with a 35 ms timeout failed in about 5–7% of runs, and an event-batching test with a 10 ms window in about 3–4%. The retries came from pytest-rerunfailures, so –reruns 2 matches Playwright’s retries: 2.

Two Retries Kept 99.2% of Buggy Builds Green

I ran the suite 5,000 times per retry setting on commit 17 — the last one without the bug — and on commit 24, the head.

Across all 20,000 runs of commit 24, the roundtrip test failed its first attempt 20.3% of the time, close to the 19.9% the shard math predicts. At two retries, 38 of the 5,000 buggy builds went red. In 981 of the 4,962 green ones, about one in five, the report flagged the roundtrip test as flaky.

The opening arithmetic assumes independent attempts, which holds for this bug because each attempt draws a new random ID: A retry after a failed first attempt failed 20.3% of the time, the same rate as first attempts. The batching test’s attempts weren’t independent: On commit 17, its retry failed 17% of the time after a failure, against about 3% for a first attempt.

An old Flake and a new one Look Different

Figure 1: A 1,200-run CI History — 24 Commits, 50 Runs Each, two Retries

Each tick is a run where that test’s first attempt failed. The roundtrip test has no failures in the first 850 runs and fails in 71 of the last 350.

Ranked by first-attempt failure rate over the whole history, the roundtrip test (5.9%) sits between the two ordinary flakes (6.7% and 3.8%). Only its onset sets it apart: The run where a test with a clean record starts failing. A dependency upgrade or a new runner image can also cause one, so an onset shows only that something changed at a known point; if that was a commit, a bisect can find it.

I replayed the history against two rules. The first is a simple automatic quarantine: Once a test’s first attempt fails in 2 of its last 50 runs, the test keeps running but no longer affects the build. It quarantined both ordinary flakes within the first 60 runs, and the roundtrip test on the second run of commit 18, after two failures in a row. From then on, the bug couldn’t turn a build red. Micco’s post names this risk: Automatic quarantine “could easily mask a real race condition or some other bug in the code being tested.”

The second rule checks onset first: A first-attempt failure after 200 consecutive runs without one marks a suspected regression, which goes to bisect instead of quarantine. It flagged the roundtrip test on the first run of commit 18 and never flagged the other two, since neither went 200 runs without a failure.

Bisecting an Intermittent Failure Takes Repeats

With a one-in-five bug, the bisect check has to run the test several times and call the commit bad if any run fails. A test that fails at rate p passes n runs in a row with probability (1 − p)ⁿ, so 95% confidence that a commit is good takes n = ln(0.05) / ln(1 − p) runs: 14 at a 20% failure rate, 59 at 5% and 299 at 1%.

With pytest-repeat installed, the check is a short script. Keep it outside the repository so checkouts don’t change it:

#!/bin/sh
# bisect-check.sh: the commit is bad if any of 14 runs fails.
# –reruns 0 overrides retries set in pytest.ini (pytest-rerunfailures).
pytest -x -q –reruns 0 –count 14 tests/test_orders.py::test_order_roundtrip
status=$?
# git bisect: 0 = good, 1-127 = bad (125 = skip), anything else stops it.
[ “$status“ -gt 1 ] && exit 128   # pytest itself failed: stop the bisect
exit “$status“

Start from a good commit where the test already exists: git bisect start HEAD <good-commit>, then git bisect run /path/to/bisect-check.sh, then git bisect reset. Without –reruns 0, retries set in pytest.ini leak into the check: With –reruns 2 there, the check run 20 times at commit 18 caught the bug twice, and with the override all 20 times.

I bisected from commit 1 to commit 24, 100 times for each repeat count:

In the one-run row, 48 bisects blamed commit 24, the head: git bisect start marks it bad without testing it and every buggy commit they tested happened to pass. Every wrong answer named a commit after 18, since a commit without the bug never failed the check.

A Quarantine Policy That Keeps Regressions Visible

I’d use this policy, with its numbers as starting points.

1. Keep test retries for the build result and record every attempt. Store each attempt’s outcome, not only the final one. Surefire’s XML keeps failed attempts as flakyFailure elements. However, pytest’s JUnit XML with pytest-rerunfailures 16.7 doesn’t: In my runs it dropped the failure, so I logged attempts with a small pytest plugin.
2. Check onset before you quarantine. When a test with a clean record starts failing, first reproduce the failure at the current commit with enough repeats. If it reproduces, treat it as a regression and bisect; if it doesn’t, quarantine it like any other flaky test. A new test needs a burn-in of repeated runs before it joins the blocking suite, or its first failure can’t be told from an onset.
3. Give every quarantine a ticket and an owner. The owner is the team that owns the code under test, not whoever’s pull request hit the failure. GitHub’s flaky-test system goes further and assigns the issue to whoever most recently changed the test or its code before the test became flaky, which uses onset to pick the person.
4. Keep quarantined tests running, and have someone watch them. Move them to a non-blocking job instead of a skip list and alert the owner when the test’s failure rate climbs or it starts failing every attempt. Slack’s first approach was this without the alerts: It filtered flaky tests’ results out of builds, then switched to disabling them with a ticket because “failing tests were leaking into the main branch.”
5. Set a deadline for each quarantined test and a cap on how many there are. I’d start with 30 days, the default suppression period in Microsoft’s Flakes system. Martin Fowler suggested a cap such as eight tests in quarantine, or a time limit such as a week, because without one “your bug detection system is eroding.” At the deadline, the owner fixes the test or deletes it. If the owner does neither, the cap is full and the next flaky test has to stay in the blocking suite.
6. Release a test only after enough consecutive passes with retries off. Size the run count with the bisect formula for the failure rate you’ll accept: 59 straight passes for 5%, 299 for 1%. Passing them shows the rate is probably below that, not that it’s zero. A partial fix that takes a test from 20% to 5% still passes 14 runs in a row about half the time. Run the passes in a loop job, and vary the test order and the machine, because some flaky tests fail only in certain orders or environments.
7. Limit whole-job retries to infrastructure failures. A job retry reruns every test in the job, and its final status hides which tests failed the first time. GitLab CI can limit it to runner problems with retry: {max: 2, when: [runner_system_failure, runner_external_dependency_failure, runner_interrupted]}. GitLab 19.1 moved some failures from the first reason into the other two.

What I Didn’t Choose, and What the Policy Costs

I didn’t recommend turning test retries off. Without them, my bug-free commit went red 7.8% of the time. Shopify made the same case for retries in its Android monorepo in 2019, and still detected every test that passed after a retry and told the developers.

I also didn’t recommend the strict switch, which fails the build on every flaky result. On a suite with my noise level, it would turn about 8–10% of builds red with no bug behind them. Kubernetes comes close: Its test jobs ‘must not automatically retry on test failures’, and its policy makes an assigned flaky test more important than anything else a contributor is doing. A strict switch needs that kind of commitment.

The policy needs per-attempt history for every test, and repeat loops cost CI minutes: Showing that a fix worked for a test that failed one run in 1,000 takes about 3,000 straight passes. The onset check also raises false alarms on rare flakes. A test that fails one run in 200 goes 200 runs without a failure about 37% of the time, so to keep false alarms near 5%, the onset rule’s window (200 runs in my replay) has to be about three times a test’s average gap between failures.

Where This Experiment Stops

This is one planted bug in a toy Python service, on one machine with one test runner. Only the random test data made the bug intermittent, so its retries were independent draws; a race condition can fail several attempts in a row. A regression in a test that’s already flaky shows up as a change in rate, say from 3% to 8%, which takes a before-and-after comparison to catch. Meta describes doing that when a test’s flakiness “regresses and starts trending higher,” and my history gave every commit 50 runs. In a real pipeline, a commit with one or two runs can merge before its test fails even once, which spreads the onset across several commits.

Start With the Tests That Began Flaking Recently

Pull as much test history from your own pipeline as you have, ideally months. List every test that passed only on a retry and sort the list by the date of each test’s first flaky result, newest first, rather than by how often it flakes. This puts the onsets at the top. Before anyone quarantines those tests, reproduce each failure with repeats and bisect it. If your reports drop failed attempts, start by recording them.