Flaky Test Passed on Retry: Why That Isn’t a Fix
A flaky test that passes on retry has not necessarily been fixed: the rerun only shows that the test passed once under different conditions. It may have hidden an intermittent defect, a test-design problem, or an unstable environment, so the cause still needs investigation.
What a flaky test is—and what a retry proves
A flaky test gives different results when run against what is intended to be the same code and test setup. It may fail on one run and pass on another without a relevant code change. The word “intended” matters: machines, timing, shared services, test data, and system state can change between runs, even when the source code does not.
A passing retry proves only that one execution completed successfully. It does not establish why the first execution failed, that the failure was harmless, or that future runs will be reliable. A retry can be useful evidence, but it is not a diagnosis.
For example, a test that checks whether a background job finishes within one second may fail on a busy runner and pass on a quiet one. The second result does not tell you whether the timeout was too strict, the job sometimes stalls, or the runner was unusually slow. Those are different problems with different remedies.
Why rerunning can conceal a real defect
Retries change the conditions under which a test runs. A later attempt may get a faster machine, a different network response, a warmed cache, a different thread schedule, or a cleaner test database. If the application has a race condition or intermittent service failure, the retry may simply land on a successful path.
Intermittent failures are still failures worth understanding. A defect that appears only under load or in a narrow timing window can affect users too. Conversely, an unreliable test may produce false alarms even when the product is working. Without examining the failure, teams cannot reliably tell these cases apart.
There is also a feedback cost. If a pipeline reports green whenever one of several attempts passes, it can train people to disregard initial failures. Important signals become routine noise, and genuinely new failures are harder to spot. A green status should communicate what the team’s policy says it means—not silently erase a failed attempt.
Common causes of flaky tests
Flakiness can come from the test, the product, or the environment. Common sources include:
- Timing assumptions: fixed sleeps, short timeouts, or assertions made before asynchronous work finishes.
- Shared state: tests using the same database records, files, ports, accounts, or mutable global data.
- Order dependence: one test leaves state behind that changes the outcome of another.
- Concurrency: race conditions, unsynchronized access, or assumptions about which task runs first.
- External dependencies: networks, remote services, clocks, queues, or third-party systems that respond variably.
- Randomness: generated data or randomized test order without a recorded seed for reproducing failures.
- Resource pressure: busy CI workers, limited memory, slow disks, or exhausted connection pools.
- Test isolation problems: cleanup that is incomplete, or tests that depend on a particular starting state.
These causes can overlap. A slow worker may expose a timing bug in the test, while shared data may make the same test pass or fail depending on execution order.
How to investigate a failure that passes on retry
Start by preserving the first failure’s evidence before another run overwrites it. Record the test name, commit, runner or environment, attempt number, timestamps, logs, and relevant artifacts such as screenshots or traces. Compare the failed and passing attempts rather than treating the final status as the whole story.
- Check what changed between attempts. Look for different workers, test order, data, environment variables, service state, or timing.
- Reproduce the conditions. Run the individual test repeatedly, then run it alongside its usual neighbors. If randomness is involved, capture and reuse the seed when possible.
- Inspect the assertion and synchronization. Ask whether the test waits for a meaningful condition or merely waits for a fixed amount of time. Verify that setup and cleanup isolate state.
- Separate product and test failures. Use logs and a smaller reproduction to determine whether the application behaves incorrectly or whether the test observes it unreliably.
- Change one likely cause and verify the change. Repeat the relevant test enough to gather evidence, and run the wider suite to catch side effects.
Repeated runs help measure how often a test fails, but a short clean streak does not prove reliability. If a test fails independently with probability p, the chance that it passes n consecutive attempts is (1 − p)n. Since the failure probability is usually unknown and may vary with conditions, even many passes are evidence—not proof—that the problem is gone.
Retries, quarantines, and durable fixes
Retries can be reasonable as a temporary resilience measure when a failure is known to be transient and the team still captures the original result. They are more informative when the system reports both the initial failure and the eventual outcome, rather than presenting a retry-pass as an ordinary clean pass.
Quarantining a test can keep a known flaky check from blocking unrelated work, but it should not mean forgetting it. Track the test, its owner, the reason for quarantine, and a plan to restore it. Otherwise, quarantine can become a permanent blind spot.
A durable fix depends on the cause. It might mean waiting for an observable completion condition instead of sleeping, giving each test isolated data, cleaning up resources, controlling randomness, repairing a race in the application, or making the test environment more consistent. Simply increasing a timeout can help when the limit is demonstrably too short, but it can also make failures slower without addressing a hang or incorrect behavior.
After changing the test or product, check both reliability and diagnostic value: does it pass consistently in the relevant environments, and will it still fail clearly when the behavior is genuinely wrong? A test made green by weakening its assertions is not necessarily a better test.
When a retry is useful—and when it misleads
A retry is useful for gathering information, recovering from a clearly understood transient infrastructure event, or reducing disruption while an investigation proceeds. It is misleading when it converts “failed, then passed” into “never failed,” or when no one examines the first attempt.
A practical policy distinguishes the initial result from the final pipeline decision. For example, a team can permit one automatic retry but retain the failure in reports, label the job as recovered, and track recurring cases. The exact policy is a team choice; the essential point is that retries should not erase evidence or substitute for ownership.
Frequently Asked Questions
Does a flaky test passing on retry mean the code is correct?
No. It means that one later execution passed. The original failure may still indicate an application defect, a test problem, or environmental instability.
How many times should I rerun a flaky test?
There is no universal number that proves a test is stable. Repeated runs can estimate how often it fails under the tested conditions, but investigation and evidence about the cause matter more than a pass count.
Should CI automatically retry failed tests?
It can, if the policy preserves and reports the first failure and recurring retries are tracked. A retry that hides failures can make the build status less trustworthy.
Is increasing a test timeout a valid fix?
Sometimes, if evidence shows the expected operation legitimately needs more time. If the test relies on a fixed delay or the operation can hang, a larger timeout may only delay the same failure.
Should a flaky test be deleted?
Not automatically. First determine whether it protects important behavior and why it is unstable. Repair it, replace it with a more reliable check, or quarantine it with an owner and follow-up plan.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.