Confidence Score: Probability, Act, or Abstain?

This article explains what a confidence score is, whether it can be treated as a true probability, and how to decide between acting on it or abstaining. The short answer: a confidence score is not automatically a probability—it only becomes one if it is explicitly calibrated—so you should act on it only when calibration is verified, and abstain when the cost of a wrong decision outweighs the expected benefit.

Formula

The expected cost of acting is:

Expected cost = p × C_FP + (1 − p) × C_FN

• p = predicted probability (your confidence score, between 0 and 1)
• C_FP = cost if you act and the prediction is wrong (false positive)
• C_FN = cost if you do nothing and the prediction was true (false negative)

If the expected cost of acting is less than the cost of abstaining (which is simply C_FN, because you miss the true case), then act; otherwise abstain.

Worked Example

Using the defaults: p = 0.8, C_FP = $100, C_FN = $500, threshold = 0.5.

1. Expected cost of acting = 0.8 × $100 + 0.2 × $500 = $80 + $100 = $180.
2. Cost of abstaining = $500.
3. Since $180 < $500, the calculator says “act� (1).

Even though the confidence score is 0.8 (not extremely high), the asymmetric costs make acting the rational choice.

Background: Where Confidence Scores Come From

Confidence scores appear everywhere: a spam filter says 0.97, a face-recognition system says 0.82, a medical diagnostic model says 0.65. These numbers are supposed to tell you how sure the system is. But “sure� can mean different things. In machine learning, a confidence score is often the raw output of a model—for example, the softmax probability of a neural network, or the distance to a decision boundary in a support vector machine. These raw scores are not inherently probabilities. They are just numbers that rank how strongly the model leans one way or another.

To turn a score into a true probability, the model must be calibrated. Calibration means that if the model says 0.8 for 100 cases, then roughly 80 of those cases are actually positive. A well-calibrated model produces confidence scores that match observed frequencies. Uncalibrated models can be overconfident (saying 0.9 when only 0.7 are true) or underconfident. The classic calibration methods are Platt scaling and isotonic regression, both of which adjust the raw scores using a validation set.

How to Tell If a Confidence Score Is a Probability

You can check calibration with a reliability diagram: bin the predictions (e.g., 0.0–0.1, 0.1–0.2, …), then plot the average predicted score against the actual fraction of positives in each bin. If the points lie near the diagonal, the score is well-calibrated. Another metric is the expected calibration error (ECE), which averages the gap between predicted and actual across bins. A low ECE (e.g., <0.05) suggests the score can be treated as a probability.

However, even a calibrated score is only a probability under the same distribution as the training data. If you apply the model to a different population, the calibration breaks.

Act or Abstain: The Decision Rule

Once you have a calibrated probability, the decision of whether to act depends on the costs. The optimal rule is to act if the expected benefit of acting exceeds the expected benefit of abstaining. The calculator above implements this with a simple cost model. In practice, you also need to consider the base rate (how often the event occurs) and any other utilities, but the principle is the same: compare expected outcomes.

When should you abstain? When the cost of a false positive is very high relative to the cost of a false negative, or when the probability is close to 0.5 and both costs are large. For example, in medical screening, a false positive leads to unnecessary biopsies (costly and risky), so you might set a high threshold (e.g., 0.9) before acting. In fraud detection, a false negative (missing fraud) is often more costly, so you might act on a lower threshold.

Common Mistakes and Misconceptions

• Treating raw scores as probabilities: A softmax output of 0.9 is not necessarily a 90% chance. It could be 70% or 95% depending on calibration.
• Ignoring the base rate: Even with a high confidence score, if the event is rare, the probability of a false positive can be surprisingly high. For example, a test with 95% accuracy on a disease that affects 1% of people yields a positive predictive value of only about 16%.
• Using a single threshold for all decisions: The optimal threshold depends on the cost ratio. A 0.5 threshold is only optimal when false positives and false negatives cost the same.
• Assuming calibration is permanent: Models drift as the world changes. Recalibrate periodically.

How to Choose a Threshold

Choose a threshold by minimizing the expected cost, or equivalently maximizing expected utility. If you don’t know the costs precisely, you can plot a cost curve or use a receiver operating characteristic (ROC) curve to see the trade-off between true positive rate and false positive rate. Pick the point that gives the best balance for your situation. If the costs are unknown, a common heuristic is to set the threshold to the base rate of the event, but that is only optimal under specific conditions.

Alternatives to a Single Confidence Score

Instead of a single number, some systems output a prediction interval or a credible interval that captures uncertainty. For example, a model might say “there is a 95% chance the true value is between 0.6 and 0.9.� This is more informative than a point estimate. Another alternative is to use conformal prediction, which produces sets of labels with a guaranteed coverage rate (e.g., 90% of the time, the true label is in the set). These methods give you a more principled way to abstain: if the set contains more than one label, you can abstain.

Safety and Ethical Considerations

In high-stakes domains (medical, legal, financial), never rely solely on a confidence score. Always have a human in the loop, especially when the cost of error is high. Also, be aware of fairness: a model may be well-calibrated overall but poorly calibrated for certain subgroups (e.g., different ages, genders, or ethnicities). Check calibration separately for each relevant group.

Frequently Asked Questions

What is the difference between confidence score and probability?

A confidence score is any numeric measure of certainty; a probability is a specific type of score that obeys the axioms of probability (0 to 1, sums to 1, etc.) and matches long-run frequencies. A confidence score only becomes a probability after calibration.

How do I calibrate a confidence score?

Use Platt scaling (fitting a logistic regression on the scores) or isotonic regression (a non-parametric monotonic mapping) on a held-out validation set. Then evaluate calibration on a separate test set.

What is a good confidence score to act on?

There is no universal number. It depends on the cost of being wrong. For low-cost decisions, 0.6 might be fine; for high-cost decisions, you might need 0.99. Use the expected cost formula to find your threshold.

Why is my model overconfident?

Overconfidence often comes from model complexity (e.g., deep neural networks), small training data, or training with a loss that doesn’t penalize overconfidence. Regularization and temperature scaling can help.

Can I use confidence scores to decide when to abstain?

Yes. A common strategy is to abstain when the confidence score falls below a certain threshold (or when the top two probabilities are too close). This is called a “reject option� in classification.

What is the expected calibration error (ECE)?

ECE is a metric that measures the average difference between predicted confidence and actual accuracy across bins. Lower is better. It’s a standard way to quantify calibration.

Related

• Volume Macintosh HD Was Found Corrupt? Fixes Explained
• WD Book Data Recovery: What It Means & How to Do It
• Samsung SRF7300SA: What to Know Before Buying

in p | Predicted probability | - | 0.8 | 0
in cost_fp | Cost of false positive | $ | 100 | 0
in cost_fn | Cost of false negative | $ | 500 | 0
in threshold | Decision threshold | - | 0.5 | 0
out expected_cost | Expected cost of acting | $ | p*cost_fp + (1-p)*cost_fn | 2
out cost_abstain | Cost of abstaining | $ | cost_fn | 2
out action | Should you act? | - | if(expected_cost < cost_abstain, 1, 0) | 0