False Positive vs Missed Case: The Rule

This article explains a rule that helps avoid false positives while still catching real cases. The rule is: set your detection threshold based on the cost of errors, not on a fixed default. You lower the threshold when missing a case is more harmful, and raise it when false alarms are more costly.

Formula

The F1 score is the harmonic mean of precision and recall:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

Where Precision = True Positives / (True Positives + False Positives) and Recall = True Positives / (True Positives + False Negatives). Both are expressed as decimals or percentages, but the formula requires them to be in the same form.

Worked Example

Using the default values: Threshold = 0.5, Precision = 90%, Recall = 80%.

F1 = 2 × (0.90 × 0.80) / (0.90 + 0.80) = 2 × 0.72 / 1.70 = 1.44 / 1.70 = 0.8471, rounded to 0.85.

So with a threshold of 0.5, you get an F1 score of 0.85, meaning a good balance between false positives and missed cases.

Why the Default Threshold Often Fails

Most classification systems use a default threshold of 0.5, meaning any probability above 50% is classified as positive. This works well when false positives and false negatives have equal costs. But in many real-world situations, the costs are not equal. For example, in medical screening, a false negative (missing a disease) can be life-threatening, while a false positive only causes anxiety and extra tests. In spam filtering, a false positive (deleting a legitimate email) is more annoying than a false negative (letting spam through).

The rule that avoids both extremes is to adjust the threshold based on the cost ratio. If a missed case costs 10 times more than a false alarm, lower the threshold to catch more cases, even if it increases false positives. If a false alarm costs 10 times more, raise the threshold.

How to Choose the Right Threshold

Here is a practical process:

• List the costs: estimate the cost of a false positive and a false negative in your specific context.
• Calculate the cost ratio: cost of false negative / cost of false positive.
• Adjust the threshold: if the ratio is high (e.g., 10), set the threshold lower (e.g., 0.3). If the ratio is low (e.g., 0.1), set it higher (e.g., 0.7).
• Validate with real data: test the chosen threshold on a hold-out dataset to see the actual precision and recall.
• Re-evaluate periodically: thresholds should be re-checked as data and costs change.

Common Mistakes and Misconceptions

A common mistake is to assume that a higher F1 score always means a better model. F1 does not consider the cost of errors, only the balance. For example, a model with precision 95% and recall 95% has an F1 of 0.95, but if a false negative is catastrophic, you might prefer a model with 90% precision and 99% recall (F1 = 0.94).

Another mistake is to use the same threshold for all subgroups. In fraud detection, the cost of a missed fraud might be higher for large transactions than small ones. Using a single threshold ignores this.

Also, people often confuse precision and recall. Precision answers “of those I flagged, how many were correct?� Recall answers “of all the real cases, how many did I catch?�

Comparing Different Rules

There are several rules for setting thresholds:

• Fixed 0.5: simple but ignores cost.
• Youden’s J statistic: maximizes (Sensitivity + Specificity – 1) on the ROC curve, which treats false positives and false negatives equally.
• Cost-based threshold: minimizes expected cost, which is the rule described here.
• Prevalence-based threshold: set the threshold to the base rate of the positive class, which works well when the cost ratio is 1.

The cost-based rule is the most flexible because it explicitly incorporates the real-world impact of errors.

Frequently Asked Questions

What is the difference between a false positive and a false negative?

A false positive is when the system predicts a case is positive (e.g., disease present) but it is actually negative. A false negative is when it predicts negative but the case is actually positive. In medical terms, a false positive is a false alarm, a false negative is a missed diagnosis.

How do I know if my threshold is too low or too high?

If you are seeing too many false positives, your threshold is too low. If you are missing too many real cases, your threshold is too high. The optimal point is where the cost of false positives equals the cost of false negatives.

Can I use the F1 score to compare models with different thresholds?

Yes, but only if the cost ratio is 1. If costs are unequal, use a cost-based metric like expected cost or a weighted F1 score.

What is the best threshold for imbalanced datasets?

For imbalanced datasets, the default 0.5 often performs poorly because the model is biased toward the majority class. Lowering the threshold helps catch more minority class cases, but you should set it based on cost, not just imbalance.

Does the rule apply to multi-class problems?

Yes, but you need to set a threshold for each class or use a one-vs-rest approach. The same cost-based principle applies: compare the cost of misclassifying each class.

Limitations and When to Use Alternatives

The cost-based threshold rule assumes you can estimate costs, which is not always possible. In some domains, costs are hard to quantify, so you might use a qualitative approach or a conservative default like 0.5. Also, the rule assumes that the model’s predicted probabilities are well-calibrated, meaning a 70% probability actually occurs 70% of the time. If the model is poorly calibrated, you should calibrate it first (e.g., using Platt scaling or isotonic regression) before setting a threshold.

For extremely imbalanced datasets, consider using anomaly detection or resampling techniques rather than just adjusting the threshold.

Related

• 5 Story-Driven Games Like Life Is Strange
• Mini LED vs OLED for Gaming: HDR, Response, Burn-In
• Show Hidden Files Mac Finder: 3 Ways

in t | Threshold (0-1) | - | 0.5 | 0
in p | Precision at threshold | % | 90 | 0
in r | Recall at threshold | % | 80 | 0
out f1 | F1 Score | - | 2 * (p/100 * r/100) / ((p/100) + (r/100)) | 2
out cost | Total cost per 100 | - | (100 - r) * 1 + (100 - p) * 1 | 2