Fix Flaky Tests From AI-Generated Test Code

Updated 5 November 2026 · Standard formula, shown below

Flaky tests pass sometimes and fail other times without a relevant code change, making builds hard to trust. This guide helps developers identify the cause in generated test code, replace fragile assumptions with deterministic checks, and verify that the suite behaves consistently. You’ll need access to the test files, the command used to run them, and the relevant application code or test documentation.

What You’ll Need

The repository and a way to run its test suite locally or in CI.
The exact failing test output, including stack trace and logs.
Any required test configuration, fixtures, and environment variables.
A clean working tree or a branch where you can make and review changes.

Step-by-Step

Confirm that the failure is actually flaky. Record the failing test, command, environment, and whether the failure happens locally, in continuous integration, or both. Rerun the smallest relevant test several times without changing code. If it fails consistently, investigate a normal defect rather than treating it as flakiness. Repeated passes do not prove a test is reliable, but repeated inconsistent results are a useful signal.
Read the failure before editing the test. Check the assertion, stack trace, captured output, and any setup or teardown logs. Identify whether the failure concerns a timeout, unexpected order, missing data, a network call, a date, or shared state. Generated tests may contain plausible-looking assertions that do not match the application’s actual contract. Compare the test with the production code, API schema, or established project examples; do not simply weaken an assertion to make the run pass.
Reduce the reproduction. Run the single test using the project’s documented test command. Then run it repeatedly, and, if relevant, run the containing test file and suite. Use the framework’s supported repeat or randomized-order options only after checking their syntax. Save the command and output so you can compare results. A failure that appears only in the full suite often points to shared state, ordering, resource contention, or cleanup problems.
Remove dependence on real time. Look for sleeps, short timeouts, current-time assertions, and assumptions about when a background task finishes. A fixed delay is not proof that work is complete: fast systems waste time, while slow ones still fail. Prefer the framework’s fake clock or controlled scheduler for time-dependent logic, and advance it explicitly. For asynchronous work, await the returned promise or use a condition-based wait with a sensible upper bound and a clear failure message. Avoid increasing a timeout as the only fix.
Control randomness and generated data. Check for random values, randomized ordering, auto-generated identifiers, and fixtures whose contents can vary. Use a fixed seed or explicit test data where supported, and report the seed when a failure occurs. Keep values representative of the behavior under test, but avoid depending on incidental formatting or implementation details. If randomness is itself the subject of the test, make its source controllable and verify the intended range or property rather than one unpredictable exact result.
Isolate external services and machine-specific assumptions. Determine whether the test calls a live API, database, filesystem, clock, or network service. Unit tests should usually replace external dependencies with controlled fakes or stubs; integration tests should use an explicitly configured test service and known fixtures. Do not rely on a developer’s local files, credentials, timezone, locale, port availability, or internet access. Restore environment variables and temporary files during teardown.
Make setup, cleanup, and ordering explicit. Check for shared mutable fixtures, reused database records, global mocks, and tests that depend on another test running first. Give each test the data it needs, use unique keys when parallel workers share a resource, and clean up even when an assertion fails. Reset mocks and global state in the framework’s standard lifecycle hooks. A test should pass when run alone and should not change what a later test observes.
Assert behavior, not incidental implementation. Generated code can overfit to internal calls, exact error wording, element positions, or timing. Prefer observable outcomes: returned values, persisted state, documented events, or accessible user-facing behavior. Keep assertions specific enough to catch a real regression, and confirm the expected behavior from requirements or existing contracts rather than accepting the generated test’s premise.
Validate the repair under varied conditions. Run the repaired test alone repeatedly, then run the relevant file and full suite, including the same parallelism used in CI. If CI is the only failing environment, compare runtime, operating system, timezone, locale, dependencies, and worker configuration. Review the final diff to ensure the change fixes the cause rather than suppressing the failure. Record any remaining environmental limitation in the test or project notes.

Common Mistakes

Adding sleeps or greatly extending timeouts: these hide scheduling problems and still fail under different load.
Retrying every failure: retries can mask real defects and make CI results misleading. Use them only when the project has a justified policy for a known transient boundary, and track the underlying cause.
Deleting or loosening assertions: a passing test that no longer checks the required behavior is not a fix.
Trusting generated setup without review: invented APIs, stale fixtures, and incorrect assumptions can be syntactically valid but semantically wrong.
Testing only in isolation: suite-only failures often reveal shared state or order dependence, so both isolated and full-suite runs matter.

Tips

Keep a short failure log with the test name, command, environment, and whether the result is reproducible.
Prefer dependency injection or existing test utilities over custom global patches.
When a test genuinely needs a live dependency, label it clearly and separate it from fast, deterministic unit tests.
Ask code-generation tools for a test plan and assumptions, then verify each assumption against project code. Review generated tests like any other code; passing output alone is not evidence of correctness.
Make failure messages explain what condition was expected and what was observed.

Troubleshooting

The test passes alone but fails in the suite: run the suite with randomized order if supported, then inspect global fixtures, shared database rows, mock cleanup, and parallel workers. Temporarily narrowing the run can help identify which neighboring test leaks state; keep the final fix in setup or teardown rather than adding an order dependency.

It fails only in CI: compare the exact command and configuration first. Check whether CI uses a different runtime, operating system, timezone, locale, dependency lockfile, worker count, or resource limit. Make assumptions explicit in test setup instead of relying on a developer machine’s defaults.

A network or service test fails intermittently: verify that the test is meant to use a live service. For unit coverage, stub the boundary and test error handling separately. For integration coverage, check service readiness, test credentials, fixture isolation, and cleanup; do not make external availability an unnoticed prerequisite for every test run.

Repeated runs never reproduce the failure: preserve the original CI logs and environment details. Look for intermittent resource pressure, parallel execution, and nondeterministic data. Add targeted diagnostics at the boundary, not excessive logging of secrets or sensitive values, then rerun with the closest available CI settings.

Frequently Asked Questions

How many times should I rerun a flaky test?
There is no universal count. Run enough times to see whether the behavior is inconsistent, and include the project’s normal CI conditions. A finite set of passes cannot guarantee the test will never flake.

Should I use retries to fix a flaky test?
Usually not as the primary fix. Retries may be appropriate for a documented transient integration boundary, but they can conceal defects and should not replace deterministic test setup.

Are AI-generated tests more likely to be flaky?
Generation alone does not determine reliability. Tests become flaky when they rely on uncontrolled time, randomness, shared state, external services, or incorrect assumptions; generated code should be reviewed for those risks.

Can I fix flakiness by making the assertion less strict?
Only if the original assertion was stricter than the documented behavior. Otherwise, changing it may hide a real regression. Establish the contract first, then assert that behavior directly.

What if only a parallel test run fails?
Check for shared resources, non-unique records, global state, and unsafe cleanup. Isolate each worker’s data or resource, and confirm the test does not depend on execution order.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.