A/B Test Duration Calculator: Estimate Test Length

An A/B test duration calculator estimates how long an experiment may need to run to detect a difference between two versions of a page, feature, or message. Enter the baseline conversion rate, the smallest conversion-rate change worth detecting, the daily eligible visitors per version, and the statistical settings; the calculator returns an estimated number of days, based on a standard two-proportion sample-size approximation.

Formula

This estimate uses a normal-approximation sample-size formula for comparing two independent conversion proportions with equal allocation. Let p be the baseline conversion rate as a proportion, and let δ be the absolute difference to detect. For a relative lift, δ = p × lift. With equal-sized groups, the approximate visitors required per version are:

n = 2 × (z_1−α/2 + z_1−β)² × p × (1 − p) ÷ δ²

• n is the required eligible visitors in each version, measured in visitors.
• p is the baseline conversion rate as a proportion, not a percentage number; 5% becomes 0.05.
• δ is the minimum detectable absolute conversion-rate difference, also a proportion. A 10% relative lift from a 5% baseline means δ = 0.005, or 0.5 percentage points.
• α is the two-sided significance level. At 5%, the normal critical value z_1−α/2 is approximately 1.96.
• β is the probability of missing a true effect of the specified size; power is 1 − β. At 80% power, z_1−β is approximately 0.8416.

The calculator uses the 1.96 and 0.8416 values shown in the expression, so its formula assumes a 5% two-sided significance level and 80% power. The power and alpha fields are included to make those concepts visible, but changing them does not change this fixed approximation. Duration in days is the required visitors per version divided by the daily eligible visitors per version, rounded up to a whole day. Visitors must be available to both versions in the stated daily amount.

Worked Example

Using the calculator’s defaults: baseline conversion rate = 5%, minimum detectable relative lift = 10%, eligible traffic = 1,000 visitors per version per day, power = 80%, and significance level = 5%.

1. Convert the baseline: p = 5 ÷ 100 = 0.05.
2. Find the absolute effect: δ = 0.05 × (10 ÷ 100) = 0.005, equal to 0.5 percentage points.
3. Calculate the sample per version: n = 2 × (1.96 + 0.8416)² × 0.05 × 0.95 ÷ 0.005² ≈ 29,836.5 visitors.
4. Estimate duration: 29,836.5 ÷ 1,000 ≈ 29.84 days; rounding up gives 30 days.

The result is an estimate, not a promise that a test will produce a conclusive answer on that date. The required-visitor output is rounded to a whole visitor for display; the duration calculation uses the unrounded formula result before rounding up.

How to Use the Estimate

Start with a baseline rate measured on the same kind of eligible audience and conversion event that the experiment will use. Define the smallest effect that would matter for the decision. The calculator treats the lift as relative: if the baseline is 5% and the lift is 10%, the target is 5.5%, not 15% and not a 10-percentage-point increase.

Enter traffic for one version, not combined traffic. If 2,000 eligible visitors arrive each day and are split evenly, enter 1,000. For an uneven allocation, or if one version receives substantially less traffic, this equal-group formula does not directly represent the design; use a method that supports the planned allocation.

Use the estimate to plan a sensible minimum run, then evaluate the result with an analysis appropriate to the experiment. Include the full time needed for visitors to become eligible and for the conversion window to mature. A purchase, for example, may happen after a visit, so counting visitors before their outcomes can be observed can make results incomplete.

What Changes Test Duration

• Smaller effects take longer. Required sample size is proportional to 1 ÷ δ². Halving the detectable difference takes about four times as many visitors, all else equal.
• Lower baseline rates can require more traffic. The formula depends on p(1 − p) as well as the absolute effect. The result also depends on whether the chosen effect is a relative or absolute lift.
• More traffic shortens calendar time. Doubling eligible daily visitors per version roughly halves the estimated days, provided traffic and assignment remain stable.
• Higher power requires more observations. More power reduces the chance of overlooking an effect of the stated size, but increases the sample needed.
• More comparisons need more care. Running many variants or checking many outcomes can raise the chance of false-positive findings unless the analysis accounts for multiple comparisons.

Common Mistakes and Limitations

A common mistake is confusing percentage points with relative percent. Moving from 5% to 5.5% is a 0.5-percentage-point change and a 10% relative lift. Since the formula squares the absolute difference, entering the wrong interpretation can dramatically alter the estimate.

Another mistake is ending a test as soon as a dashboard displays a favorable result. Repeatedly checking ordinary significance tests and stopping whenever a result looks good can inflate false-positive risk. Set the sample and analysis plan in advance, or use a sequential method designed for ongoing monitoring. Do not treat the calculator’s day count as a statistical stopping rule by itself.

This approximation assumes independent visitor outcomes, stable traffic, equal allocation, a binary conversion outcome, and a normal approximation to the sampling distribution. It does not account for clustering (such as multiple observations from one person), changing traffic mix, bots, seasonality, implementation errors, missing outcomes, or a delayed conversion window. Very low conversion rates or small samples may call for an exact or otherwise more suitable method. The estimate also does not determine whether an observed effect is practically valuable.

Frequently Asked Questions

How long should an A/B test run?
Run it long enough to reach a preplanned sample size and observe the relevant outcome window. The calendar estimate depends on the effect size, baseline rate, power, significance level, and eligible traffic.

What is a good minimum detectable effect?
Choose the smallest change that would matter for the decision, rather than selecting a convenient number. A smaller detectable effect needs more observations, so the choice should reflect both practical importance and available traffic.

Does this calculator use relative lift or percentage points?
Its lift input is relative. It converts that value to an absolute difference using the baseline rate. For a 5% baseline and 10% relative lift, the absolute difference is 0.5 percentage points.

Can I stop the test when the result becomes significant?
Not safely with an ordinary fixed-sample analysis if you repeatedly inspect results and stop at the first significant reading. Plan a sample in advance or use a valid sequential-testing approach.

Why did my actual test take longer than the estimate?
Actual eligible traffic may be lower or uneven, conversions may need time to mature, or the experiment may have operational disruptions. The calculator also uses simplifying assumptions and gives an estimate rather than a guaranteed completion time.

Related

• 5 Number Summary Calculator: Find the Five Key Values
• Binomial Probability Calculator
• Calculate the Standard Deviation

in base | Baseline conversion rate | % | 5 | 0
in lift | Minimum detectable relative lift | % | 10 | 0
in traffic | Eligible visitors per version per day | visitors/day | 1000 | 0
in power | Statistical power | % | 80 | 0
in alpha | Significance level | % | 5 | 0
out days | Estimated duration | days | ceil((2 * (1.96 + 0.8416)^2 * (base / 100) * (1 - base / 100) / (((base / 100) * (lift / 100))^2)) / traffic) | 0
out visitors | Required visitors per version | visitors | 2 * (1.96 + 0.8416)^2 * (base / 100) * (1 - base / 100) / (((base / 100) * (lift / 100))^2) | 0