Green Test Run: Evidence Not Proof for 4 Coding Agent Myths
A green test run—where all automated checks pass—is strong evidence that a coding agent’s changes work as intended, but it is not proof. This distinction matters because four common myths about coding agents collapse when you understand what a green test actually does and does not verify. This article explains why a passing test suite is necessary but insufficient, and what that means for how you should evaluate AI coding tools.
What a Green Test Run Actually Verifies
A green test run means every test in the suite executed and returned a passing result. That is a real, useful signal: the code compiles, the functions return expected outputs for the tested inputs, and no assertions failed. For a coding agent, a green run typically follows an edit to source files, and the agent often runs the tests itself to confirm its patch did not break existing behavior.
However, the scope of that verification is strictly limited to the cases the tests cover. If a test suite has 50 tests, a green run proves those 50 scenarios work. It says nothing about the 5,000 scenarios not in the suite. This is the foundation for understanding why green runs cannot prove broader claims about agent quality.
Myth 1: Green Tests Mean the Code Is Correct
The most pervasive myth is that a passing test suite equals correct code. In reality, tests are only as good as the assertions they make. A test can pass while the underlying logic is wrong if the expected value is also wrong, or if the test only checks a happy path.
Consider a function that calculates shipping cost. A test might assert that a 5kg package costs $10. If the agent changes the function to always return $10, that test passes, but the function is now incorrect for every other weight. The green run gives false confidence because the test suite lacks boundary cases, negative cases, or property-based checks. A green run proves consistency with the tests, not correctness against the specification.
Myth 2: Green Tests Mean No Regression
A second myth is that a green run guarantees no existing functionality broke. This is only true if the test suite has full coverage of all previous behavior—which is rare in practice. Most projects have gaps: untested edge cases, untested UI flows, or untested integration points.
For example, an agent might modify a database query to improve performance. The unit tests for that query pass, but the application’s login flow, which depends on that query, breaks because the agent changed the return type. If no test covers the login flow, the green run misses the regression entirely. A green run only proves that the tested behaviors remain intact, not all behaviors.
Myth 3: Green Tests Mean the Agent Understood the Task
Many assume that if the agent’s changes pass tests, the agent understood the user’s request. This conflates output with comprehension. An agent can produce a passing result by pattern-matching on similar code in its training data, by overfitting to the specific tests, or by taking a shortcut that happens to work for the given inputs.
For instance, a user asks the agent to “sort a list of numbers.” The agent writes a function that calls the built-in sort() method. Tests pass. But the user’s actual need was to sort a list of objects by a custom key, which the agent ignored. The green run validates the code, not the interpretation of the request. A human reviewer must still check that the agent solved the right problem.
Myth 4: Green Tests Mean the Agent Is Reliable
The fourth myth is that a single green run predicts future performance. Reliability is a statistical property across many tasks, not a one-off outcome. An agent might pass tests on a simple refactoring task but fail on a complex multi-file change. A green run on one task tells you nothing about the distribution of outcomes on other tasks.
To evaluate reliability, you need a benchmark suite with hundreds of diverse tasks, run multiple times to account for randomness in the agent’s sampling. A single green run is an anecdote, not a dataset. This is why claims like “this agent passes 80% of tasks” require a controlled evaluation, not just one successful run.
How to Properly Evaluate a Coding Agent
Given that green runs are evidence but not proof, how should you evaluate an agent? Use a combination of signals:
- Test coverage analysis: Check what percentage of lines and branches the test suite covers. Higher coverage reduces the chance of missed regressions, but does not eliminate it.
- Manual code review: Read the diff yourself. Look for logic errors, missing edge cases, and deviations from the request.
- Run additional tests: Add your own tests for scenarios the agent might have missed, especially boundary values and error conditions.
- Check integration: Run the full application or a staging environment to verify end-to-end behavior, not just unit tests.
- Repeat tasks: Run the same task multiple times to see if the agent is consistent or if it gets lucky sometimes.
- Use a benchmark: If available, compare the agent against a known benchmark like HumanEval or SWE-bench for a broader reliability estimate.
These steps turn a single green run into a more robust evidence base. The key is to treat the green run as a necessary condition, not a sufficient one.
Common Pitfalls When Interpreting Green Runs
Even experienced developers misinterpret green runs. Common pitfalls include:
- Trusting coverage percentages blindly: 80% line coverage sounds good, but the uncovered 20% might contain the most critical logic.
- Ignoring test quality: Tests that assert trivial properties or that are duplicated add little value. A green run on weak tests is weak evidence.
- Forgetting about flaky tests: If tests are flaky, a green run might be a false positive. Check if the tests pass consistently across multiple runs.
- Assuming the agent ran the right tests: Some agents might run only a subset of tests. Verify that the full suite was executed.
Being aware of these pitfalls helps you avoid overvaluing a green run. Always ask: what exactly did the tests check, and what did they not check?
Frequently Asked Questions
If tests pass, can I merge the agent’s code without review?
No. A green test run is a necessary check, but it does not replace human review. The tests might be incomplete, the code might not match the intent, or the change might introduce subtle issues that tests don’t catch. Always review the diff and run additional checks before merging.
How many tests do I need for a green run to be meaningful?
There is no magic number. What matters is coverage of the behaviors that matter for your project. Aim for high branch coverage on critical modules, and include tests for edge cases, error handling, and integration points. A suite of 20 well-designed tests is more meaningful than 200 tests that all check the same happy path.
Can a green run ever be considered proof?
Only in trivial cases where the test suite is exhaustive, which is practically impossible for any non-trivial program. Formal verification can provide proof, but that is a different, much more expensive process. For everyday development, treat green runs as strong evidence, not proof.
What if the agent writes its own tests and they pass?
Be extra cautious. An agent writing tests that pass is circular reasoning—it can generate tests that match its own implementation, even if that implementation is wrong. Review the tests the agent wrote to ensure they are meaningful and not just tautological.
Does a red test run prove the agent is bad?
Not necessarily. A red run could be due to a pre-existing failing test, a flaky test, or an environment issue unrelated to the agent’s change. Check the failure messages and see if the agent’s changes caused the failure. A red run is a signal to investigate, not a verdict.
How do I know if the agent actually ran the tests?
Most agents log their actions. Look for the test command in the agent’s output or logs. If you are unsure, run the tests yourself after the agent finishes. This also verifies that the environment is consistent.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.