A/B Testing Significance Calculator: How to Read Results

An A/B testing significance calculator estimates whether the difference between two conversion rates is unlikely to be explained by random sampling variation alone. Enter the number of visitors and conversions for each version; the calculator returns a two-sided p-value and an approximate 95% confidence interval for the difference.

Formula

This calculator uses a pooled two-proportion z-test. Let n_A and n_B be visitors in each group, and x_A and x_B their conversions. The observed rates are p_A=x_A/n_A and p_B=x_B/n_B. The headline difference is 100(p_B−p_A) percentage points.

For the null hypothesis that the rates are equal, the pooled rate is p=(x_A+x_B)/(n_A+n_B). The z statistic is (p_B−p_A)/√[p(1−p)(1/n_A+1/n_B)]. The two-sided p-value is 2[1−Φ(|z|)], where Φ is the standard normal cumulative distribution function. The formula block approximates this function numerically.

The confidence interval uses the unpooled standard error: (p_B−p_A) ± 1.96√[p_A(1−p_A)/n_A+p_B(1−p_B)/n_B]. Its endpoints are reported in percentage points. A confidence interval that excludes zero aligns approximately with a two-sided test below 0.05, subject to the test assumptions and rounding.

Worked Example

Using the default entries, version A has 100 conversions among 1,000 visitors, so its rate is 0.10, or 10%. Version B has 120 among 1,000, or 12%. The difference is 12%−10%=2.00 percentage points.

• Pooled rate: (100+120)/(1,000+1,000)=0.11.
• Standard error under the null: √[0.11×0.89×(1/1,000+1/1,000)] ≈ 0.01399.
• z: (0.12−0.10)/0.01399 ≈ 1.43.
• Two-sided p-value: 2[1−Φ(1.43)] ≈ 0.1535.
• Unpooled interval: 0.02 ± 1.96√[0.10×0.90/1,000+0.12×0.88/1,000], which is approximately −0.85 to 4.85 percentage points.

So the calculator displays a 2.00-point observed lift, p≈0.1535, and an approximate interval from −0.85 to 4.85 points. The data do not meet the conventional 0.05 significance threshold. That does not prove the versions are identical; the interval still allows effects in either direction.

What Statistical Significance Means

A p-value is the probability, assuming equal underlying conversion rates and the test’s assumptions, of observing a difference at least as extreme as the one measured. A p-value of 0.05 is not a 95% probability that B is better, nor a measure of how large or valuable the effect is. The conversion-rate difference and interval help convey magnitude and uncertainty; practical importance depends on the experiment’s goals and context.

“Significant” is not the same as “important.” A tiny difference can become statistically detectable with a very large sample, while a potentially useful difference may remain uncertain in a small sample. Look at the absolute rates, the percentage-point change, the interval, and the outcome’s relevance—not just a threshold.

When to Use This Calculator

Use it to make a quick comparison of one binary outcome, such as whether a visitor completed a signup or purchase, between two independent groups. The entries must use the same definition of visitor and conversion for both versions. Each visitor should contribute to one group and have one outcome for the measured event.

Run the experiment with a plan: choose the primary metric, decide how long data collection should run, and estimate the sample size needed to detect a meaningful difference. Stop rules matter. Checking significance repeatedly and stopping as soon as the p-value falls below 0.05 can inflate the chance of a false positive; ordinary fixed-sample z-test results do not correct for that practice.

Assumptions and Common Problems

• Independent observations: repeated visits from the same person, household, or account can violate the simple model if treated as separate independent visitors.
• Random assignment: differences in audience, traffic source, device mix, or timing can confound the comparison if assignment is not properly randomized.
• Enough expected outcomes: the normal approximation may be unreliable with sparse data, very low conversion rates, or zero conversions. Consider an exact test or a method designed for sparse counts.
• Correct denominators: enter eligible visitors, not page views or sessions, unless those are truly the defined independent experimental units.
• One primary comparison: testing many metrics, segments, or variants raises the probability of finding a chance result. Predefine primary outcomes or use an appropriate multiple-comparison method.

Counts should be whole numbers, conversions cannot exceed visitors, and visitor counts must be positive. Do not round the input rates before calculating: use the original counts. The calculator rounds displayed outputs, so values close to a decision threshold should be checked with more precision and a suitable statistical tool.

Frequently Asked Questions

What p-value is statistically significant?
A common convention is p<0.05, but the threshold should be set before reviewing results and interpreted alongside the effect size, interval, study design, and number of comparisons. It is not a universal guarantee of a real or useful effect.

Does a significant result mean version B is better?
Not automatically. Check the direction of the measured difference: a positive B-minus-A value favors B for a metric where higher is better. For outcomes such as errors or cancellations, lower may be preferable. Statistical significance alone does not establish practical value or causation if assignment was flawed.

Can I use this for revenue or average order value?
Not directly. This calculator treats each observation as a binary conversion and tests proportions. Continuous outcomes such as revenue per visitor need a method suited to their distribution, often with attention to skew, outliers, and repeated observations.

What if the p-value is above 0.05?
It means the data did not provide evidence against equal rates at that chosen threshold; it does not show that the variants are equivalent. Use the confidence interval to see which effect sizes remain plausible, and consider whether the test had enough data to detect a worthwhile difference.

Why do other calculators give slightly different answers?
They may use a different test, continuity correction, exact method, one-sided alternative, or confidence-interval formula. This page uses a two-sided pooled z-test for the p-value and a standard unpooled normal-approximation interval, so results can differ slightly from other methods.

Related

• Calculate the Standard Deviation
• 5 Number Summary Calculator: Find the Five Key Values
• Binomial Probability Calculator

in a | Visitors, version A | people | 1000 | 1
in ca | Conversions, version A | people | 100 | 0
in b | Visitors, version B | people | 1000 | 1
in cb | Conversions, version B | people | 120 | 0
out diff | Conversion-rate difference, B minus A | percentage points | 100*(cb/b-ca/a) | 2
out z | Pooled z statistic | - | (cb/b-ca/a)/sqrt(((ca+cb)/(a+b))*(1-(ca+cb)/(a+b))*(1/a+1/b)) | 2
out p | Two-sided p-value | - | 2*(1-0.5*(1+abs(z)/sqrt(1+0.2316419*abs(z))*(0.3193815+(-0.3565638+1.781478*(1.3302744-1.821256*abs(z)/(1+0.2316419*abs(z)))*abs(z)/(1+0.2316419*abs(z)))*abs(z)/(1+0.2316419*abs(z))))*exp(-z^2/2)/sqrt(2*pi)) | 4
out ci | Approximate 95% interval for difference | percentage points | 100*((cb/b-ca/a)-1.96*sqrt((ca/b^2)+(ca^2/a^3)+(cb/b^2)+(cb^2/b^3))) | 2