Confidence Intervals and P-Values: What They Mean

Confidence intervals and p-values are two statistical tools that help you decide whether a result from a sample is likely to reflect a real effect in the larger population. A confidence interval gives a range of plausible values for an unknown population parameter, while a p-value tells you how compatible your data are with a specific null hypothesis. Both are used in hypothesis testing, but they answer slightly different questions and are best used together.

Formula

The confidence interval for a population mean (when the population standard deviation is known or the sample is large) is:

CI = sample mean ± (critical value) × (standard error)

Where:

• Sample mean (t) – the average of your sample data.
• Critical value – for a 95% confidence level, this is approximately 1.96 (from the standard normal distribution). For other confidence levels, use the corresponding z-score (e.g., 90% → 1.645, 99% → 2.576).
• Standard error (SE) = sample standard deviation (s) divided by the square root of the sample size (n).

The p-value for a two-tailed test about a mean is calculated from the z-score:

z = (sample mean – null hypothesis value) / standard error

Then the p-value = 2 × (1 – Φ(|z|)), where Φ is the cumulative distribution function of the standard normal distribution. The p-value is the probability of observing a sample mean at least as extreme as yours, assuming the null hypothesis is true.

Worked Example

Suppose you have a sample of 30 students, and their average test score is 50 with a standard deviation of 10. You want a 95% confidence interval for the true mean score, and you want to test the null hypothesis that the true mean is 48.

Step 1: Calculate the standard error: SE = 10 / √30 ≈ 1.826.

Step 2: Find the critical value for 95% confidence: z* = 1.96.

Step 3: Margin of error = 1.96 × 1.826 ≈ 3.58.

Step 4: Confidence interval = 50 ± 3.58, which gives (46.42, 53.58).

Step 5: Calculate the z-score for the null hypothesis: z = (50 – 48) / 1.826 ≈ 1.095.

Step 6: Two-tailed p-value = 2 × (1 – Φ(1.095)) ≈ 2 × (1 – 0.863) = 0.274.

So the 95% confidence interval is (46.42, 53.58), and the p-value is about 0.274. Since the p-value is above 0.05, you would not reject the null hypothesis at the 5% significance level.

What a Confidence Interval Tells You

A confidence interval provides a range of values that likely contains the true population parameter (like the mean). The confidence level (e.g., 95%) is the long-run proportion of times that such intervals would capture the true value if you repeated the study many times. It is not the probability that the true value lies in a specific interval – the true value is fixed, and the interval either contains it or not. The “95%” refers to the method’s reliability, not the current interval.

Confidence intervals are useful because they give both a point estimate (the sample mean) and a measure of uncertainty (the width). A narrow interval indicates a precise estimate; a wide interval suggests more variability or a small sample size.

What a P-Value Tells You

A p-value is the probability of obtaining a test statistic as extreme as the one observed, assuming the null hypothesis is true. It is not the probability that the null hypothesis is true, nor the probability that your alternative hypothesis is true. A small p-value (typically below 0.05) suggests that the observed data are unlikely under the null hypothesis, leading researchers to reject the null. A large p-value means the data are consistent with the null, but it does not prove the null is true.

P-values depend on sample size and effect size. With a very large sample, even tiny, practically insignificant effects can produce small p-values. Conversely, with a small sample, a large effect might not reach statistical significance.

How to Choose Between Them

You don’t have to choose – they answer different questions. Use the confidence interval to estimate the size of an effect and its uncertainty. Use the p-value to make a binary decision about whether the effect is statistically significant. In practice, report both: the p-value tells you if there is evidence against the null, and the confidence interval tells you how precise that evidence is.

For example, if you’re testing a new drug, the p-value might tell you it works better than placebo, but the confidence interval tells you how much better (e.g., 2 to 5 points on a scale). That range is crucial for clinical decisions.

Common Misconceptions

• “95% confidence means there’s a 95% chance the true mean is in this interval.” Wrong – the interval is fixed; the probability is about the procedure, not the interval.
• “p-value is the probability that the null hypothesis is true.” Wrong – it’s the probability of the data given the null, not the probability of the null given the data.
• “p < 0.05 means the effect is large or important.” Not necessarily – statistical significance is not the same as practical significance.
• “If p > 0.05, there is no effect.” It only means the evidence is insufficient, not that the effect is zero.

Limitations and Alternatives

Both tools rely on assumptions: random sampling, independence, and (for the formulas above) a normal distribution or a large sample size. If these are violated, the results may be misleading. Alternatives include Bayesian credible intervals (which do give a probability about the parameter), effect size measures, and equivalence testing. Also, the choice of confidence level (90%, 95%, 99%) is arbitrary; 95% is conventional but not magical.

Remember that confidence intervals and p-values are tools for statistical inference, not definitive proof. They should be interpreted in the context of the study design, potential biases, and the magnitude of the effect.

Frequently Asked Questions

What is the difference between a confidence interval and a p-value?

A confidence interval estimates a range of plausible values for the population parameter, giving you a sense of precision. A p-value tests a specific null hypothesis and gives you the probability of seeing your data (or more extreme) if that hypothesis were true. They are related: if a 95% confidence interval does not contain the null value, the p-value will be less than 0.05.

Why is a 95% confidence interval so common?

The 95% level is a convention that balances precision and reliability. It corresponds to a 5% significance level, which is a common threshold for rejecting the null hypothesis. Other levels (90%, 99%) are used when you want more or less certainty.

Can a p-value be exactly zero?

No. A p-value is a probability, so it can get very close to zero (e.g., 0.0001) but never exactly zero, because there is always some chance, however tiny, of observing extreme data under the null hypothesis. In practice, very small p-values are reported as “p < 0.001” rather than zero.

What does a wide confidence interval mean?

A wide interval indicates more uncertainty about the true parameter. It often results from a small sample size, high variability in the data, or a high confidence level. It suggests that the estimate is not very precise.

Should I report the p-value or the confidence interval?

Report both. The p-value gives the significance test result, and the confidence interval provides the range of plausible effect sizes. Many journals and guidelines recommend reporting confidence intervals, as they convey more information than a single p-value.

Related

• Steam Error Adding Friends? Fixes That Work
• Samsung Odin Mode vs Recovery Mode
• Peptide Vehicles for Cas9 Delivery: Key Findings & Limits

in t | Sample mean | - | 50 | 0
in s | Sample SD | - | 10 | 0
in n | Sample size | - | 30 | 1
in cl | Confidence level (%) | - | 95 | 50
in h0 | Null hypothesis value | - | 48 | 0
out ci_low | Lower bound | - | t - (cl/100 critical z) * (s / sqrt(n)) | 2
out ci_high | Upper bound | - | t + (cl/100 critical z) * (s / sqrt(n)) | 2
out se | Standard error | - | s / sqrt(n) | 2
out z | Z-score | - | (t - h0) / (s / sqrt(n)) | 2
out p | Two-tailed p-value | - | 2 * (1 - normalCDF(abs(z))) | 4