Evaluate an AI Code Reviewer With Known Defects

Updated 8 November 2026 · Standard formula, shown below

Evaluating an AI code reviewer with known defects means checking whether it catches deliberately planted problems without flooding reviewers with incorrect warnings. You will prepare a controlled set of code changes with confirmed defects, run the tool consistently, and compare its findings with a human-verified answer key.

What You’ll Need

The code reviewer you want to assess, configured in a test repository or isolated environment.
A small benchmark of code changes containing documented defects and clean examples, plus an answer key.
A safe, non-production repository and permission to run the tool on its contents.
A tracking sheet or simple test log for expected findings, reported findings, severity, and review time.
Optionally, a second reviewer to check the benchmark and help resolve ambiguous results.

Step-by-Step

Define what “good” means for your use case. List the defect types the reviewer is expected to catch, such as incorrect boundary handling, unsafe input processing, resource leaks, or broken error handling. Set a scope: supported languages, file types, and whether you expect findings on a diff only or on a whole repository. Decide how you will weigh missed serious defects against noisy, low-value comments before seeing results.
Build a representative benchmark. Use small, isolated code changes that each test one principal defect. Include realistic cases from your own codebase where possible, but remove sensitive data and avoid copying vulnerable production code into an exposed service. Add clean changes and harmless edge cases to test whether the tool can refrain from raising an issue. Keep examples varied enough to avoid measuring only one coding style.
Verify every expected result independently. For each example, record the relevant lines, defect description, why it matters, and the minimum evidence that would count as a correct finding. Have a knowledgeable person confirm that the defect is real and that the expected fix or behavior is appropriate. Remove cases that depend on unstated assumptions, unavailable context, or disputed style preferences. Keep the answer key separate from the files the reviewer sees.
Freeze the test conditions. Record the reviewer’s version or configuration, language settings, prompt or rules, model settings if exposed, and the date. Use the same settings for every run. Turn off unrelated checks only when they would interfere with the evaluation, and document the change. Do not tune the configuration after seeing individual examples unless you are starting a separately labeled test.
Run a baseline on clean code. Submit the clean examples first and log all findings. This gives you a measure of false alarms and shows whether repository setup, generated files, or formatting are confusing the tool. If it reports a defect on a clean example, verify the code and the warning before counting it as a false positive; the example itself may have an overlooked issue.
Run the defect examples in a consistent way. Present each change in the format the tool is meant to review, such as a pull request diff, and provide only the context it would ordinarily receive. Avoid asking leading questions that reveal the planted defect. Save the complete output, including file, line, explanation, and severity. Repeat a subset of cases to check whether results vary between runs.
Match findings to the answer key. Count a detection only if it identifies the underlying issue with enough location and explanation to help a developer act. A vague warning in the same file is not automatically a match. Count duplicate comments about one defect as one detection, and record unrelated or inaccurate claims as false positives. Note findings that are technically correct but irrelevant to the changed code separately.
Calculate and interpret basic measures. Report the number of confirmed defects found out of the number planted, and the number of incorrect findings on clean or defect examples. You can also calculate precision (correct findings divided by all findings) and recall (correctly detected planted defects divided by all planted defects). Show the raw counts beside percentages, especially for a small benchmark, and break results down by defect type and severity.
Review failures and repeat carefully. Inspect misses to see whether the defect was outside the tool’s stated scope, hidden by missing context, or simply not recognized. Inspect false alarms for recurring patterns. If you change rules or settings, rerun the complete benchmark and report the new configuration as a separate result. Keep the benchmark fixed long enough to compare versions, then refresh it to reduce the chance of overfitting.
Make a deployment decision with limits stated. Decide whether the reviewer is suitable as a helpful first pass, whether particular checks need human verification, or whether it should not be used for the tested task. Share the benchmark, method, configuration, raw counts, and known limitations with the result. Do not treat a successful test as proof that future code is safe or that human review can be skipped.

Common Mistakes

Using an unverified answer key: Incorrect labels can make a capable reviewer look bad or hide genuine misses. Confirm examples before testing.
Testing only defects: Without clean cases, you cannot judge how often the tool raises unnecessary warnings.
Counting keyword overlap as detection: A comment that names a nearby function but misunderstands the problem is not a useful catch.
Changing settings mid-test: This makes results difficult to compare. Treat each configuration as its own evaluation.
Overstating a small sample: A few examples are useful for finding obvious gaps, but percentages from them are unstable and do not establish broad reliability.

Tips

Use one principal defect per test case so a correct finding can be matched clearly.
Include realistic context, but keep examples small enough to diagnose misses.
Track severity and usefulness, not just whether a comment appeared.
Preserve raw outputs and use a consistent scoring rubric so another person can reproduce the assessment.
Separate benchmark maintenance from tool tuning; hold out some examples for a final check after changes.

Troubleshooting

The reviewer produces no output: Confirm that the test change is in the expected branch or diff, the file type is supported, and the tool has access to the relevant files. Check its run logs and configuration before concluding that it missed the defect.

It reports a different line or wording: Match by the underlying defect, not exact phrasing. Accept the finding only if its location and explanation make the issue identifiable without guesswork.

Results vary between runs: Save each output, repeat the same cases under unchanged settings, and report the variation. If repeatability matters, assess both the typical result and the worst observed result rather than selecting the best run.

Too many warnings obscure useful ones: Classify false positives by cause, then make a documented configuration change and rerun the full set. Do not remove inconvenient examples or silently change the scoring rule.

Reviewers disagree about a case: Mark it ambiguous, record the competing interpretations, and exclude it from primary scores until the expected behavior is resolved. You can still report it separately as a context-dependent case.

Frequently Asked Questions

How many examples should I use?
Use enough to cover the defect types and clean-code situations relevant to your work. There is no universal minimum; show raw counts and treat small benchmarks as exploratory.

Should planted defects be realistic or obvious?
Include both, but prioritize realistic issues that reflect your code and review tasks. Very obvious examples are useful as setup checks, not as the whole evaluation.

What counts as a false positive?
A finding is a false positive when the claimed defect is not present under the documented assumptions. Track merely low-priority or repetitive findings separately if they are technically correct.

Can I use this test to approve the reviewer for production?
It can inform a limited deployment decision, but it cannot prove safety across all code, languages, or future changes. Keep human review and other quality controls in place.

Should I tell the reviewer what defects are planted?
Not during the primary run. Provide only normal review context, then use a separate diagnostic run with hints if you want to investigate why it missed a case.

When should I rerun the evaluation?
Rerun after a meaningful reviewer, configuration, or workflow change, and periodically as the benchmark evolves. Keep a fixed subset for comparison and add new cases to limit overfitting.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.