False Positive Budget: How to Set One
This guide explains how to set a false positive budget for a statistical test—the maximum probability of rejecting a true null hypothesis that you are willing to accept. You’ll learn to define the budget, adjust it for multiple comparisons, and document it before running your analysis. You need a basic understanding of hypothesis testing and p-values.
What You’ll Need
- The number of tests you plan to run (or an estimate).
- Your chosen significance level (commonly 0.05).
- Knowledge of whether tests are independent or correlated (for some correction methods).
- A spreadsheet or calculator for basic arithmetic.
Step-by-Step
- Decide on your overall false positive limit. This is the probability of making at least one false positive across all tests. Common choices are 0.05 (5%) or 0.01 (1%). Write it down.
- Count your tests. Determine the number of comparisons you will perform. If you are unsure, use the maximum plausible number. For example, if you plan to test 20 variables, that’s 20 tests.
- Adjust for multiple comparisons (if needed). If you are running more than one test, the chance of a false positive increases. Use one of these methods:
- Bonferroni correction: Divide your overall limit by the number of tests. For 20 tests and a 5% overall limit, each test’s budget is 0.05 / 20 = 0.0025.
- Holm-Bonferroni: Rank p-values from smallest to largest, then compare each to alpha / (number of remaining tests). This is less conservative than Bonferroni.
- Benjamini-Hochberg (for false discovery rate): Control the expected proportion of false positives among rejected tests. This is often used in genomics and other high-throughput fields.
- Choose your per-test threshold. This is the p-value below which you will reject the null hypothesis. For a single test, it equals your overall limit. For multiple tests, use the corrected value from step 3.
- Document your budget. Write down the overall limit, the number of tests, the correction method, and the per-test threshold. Include this in your analysis plan or pre-registration.
- Run your tests and compare p-values to the threshold. Reject the null only when the p-value is less than or equal to your per-test threshold.
- Report the budget. In your results, state the overall budget, the correction used, and the threshold. This helps readers interpret your findings.
Common Mistakes
- Using 0.05 for every test without correction. This inflates the overall false positive rate.
- Forgetting to count all tests. Post-hoc analyses, subgroup analyses, and multiple outcome measures all count.
- Applying corrections when not needed. If you only run one test, no adjustment is necessary.
- Confusing false positive rate with false discovery rate. They are different concepts; choose the one that matches your research question.
Tips
- Set your budget before looking at the data to avoid bias.
- If you have correlated tests (e.g., repeated measures), consider using a method that accounts for correlation, such as the Westfall-Young procedure, or use a slightly more conservative correction.
- For exploratory analyses, you might set a higher false positive budget (e.g., 0.10) but clearly label the results as exploratory.
- Use a calculator or statistical software to automate the correction—many packages have built-in functions.
Troubleshooting
What if the corrected threshold is extremely small (e.g., 0.0001)? That’s expected with many tests. Consider whether you really need that many tests, or whether you can reduce the number of hypotheses. Alternatively, switch to a false discovery rate method, which is less stringent.
What if you discover more tests after setting the budget? Recalculate the correction with the new total. If you’ve already run some tests, you may need to adjust your conclusions.
What if your p-values are all above the threshold? That’s a valid result—it means no significant findings. Do not lower the threshold after the fact.
Frequently Asked Questions
What is a false positive budget?
It’s the maximum probability of incorrectly rejecting a true null hypothesis (a type I error) that you are willing to accept across all tests.
Why not just use p < 0.05 for every test?
Because with many tests, the chance of at least one false positive grows. For 20 independent tests, the probability of at least one false positive at alpha=0.05 is about 1 – (0.95)^20 ≈ 0.64, or 64%.
What is the difference between false positive rate and false discovery rate?
False positive rate (FPR) is the probability of a false positive among all tests. False discovery rate (FDR) is the expected proportion of false positives among rejected tests. FDR is often used in exploratory research.
How do I choose between Bonferroni and Benjamini-Hochberg?
Use Bonferroni when you want to control the family-wise error rate (the chance of any false positive). Use Benjamini-Hochberg when you prefer to control the false discovery rate and are okay with a small proportion of false positives among significant results.
Can I set a different budget for different tests?
Yes, but you must account for the overall error rate. For example, you could allocate more budget to a primary hypothesis and less to secondary ones, as long as the total remains within your overall limit.
What if my tests are not independent?
Correlated tests reduce the effective number of comparisons, making Bonferroni overly conservative. Consider using a method that accounts for correlation, such as permutation-based procedures.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.