How Long Should an A/B Test Run? Evidence Guide

An A/B test should run long enough to reach statistical significance with adequate power, typically 2-4 weeks for most websites, but the exact duration depends on your traffic, effect size, and desired confidence level. This calculator helps you estimate the minimum sample size needed per variant, which you can then convert into a running time based on your daily visitor count.

Formula

The sample size formula for comparing two proportions (the classic A/B test) is:

n = ( (Z_α/2 + Z_β)² × [ p₁(1-p₁) + p₂(1-p₂) ] ) / (p₂ – p₁)²

• n = required sample size per variant (number of visitors in each group)
• p₁ = baseline conversion rate (as a decimal, e.g., 0.05 for 5%)
• p₂ = expected conversion rate with the minimum effect (p₁ + MDE)
• Z_α/2 = Z-score for the significance level (1.96 for α=0.05)
• Z_β = Z-score for the power (0.84 for 80% power)

The calculator uses the same formula internally, with the effect size expressed as a relative percentage of the baseline. The output is the number of visitors needed in each variant (control and treatment) to reliably detect the specified effect.

Worked Example

Suppose your current checkout page converts at 5% (baseline), and you want to detect a 10% relative improvement (so the new version would convert at 5.5%). You choose the standard 80% power and 5% significance level.

1. Convert percentages to decimals: p₁ = 0.05, MDE = 0.10 × 0.05 = 0.005, so p₂ = 0.055.
2. Compute the standard deviation terms: p₁(1-p₁) = 0.05 × 0.95 = 0.0475; p₂(1-p₂) = 0.055 × 0.945 = 0.051975.
3. Sum them: 0.0475 + 0.051975 = 0.099475.
4. Z-scores: Z_α/2 = 1.96, Z_β = 0.84, sum = 2.80, square = 7.84.
5. Numerator: 7.84 × 0.099475 = 0.779884.
6. Denominator: (0.055 – 0.05)² = 0.000025.
7. n = 0.779884 / 0.000025 = 31,195.36 → round up to 31,196 visitors per variant.

The calculator returns 31,196 (rounded up to the nearest whole number). If you get 2,000 visitors per day to the page, you would need 31,196 / 2,000 ≈ 16 days per variant, so about 16 days total (since both variants run simultaneously).

Why “Not Weeks� – The Real Determinants

The common advice to “run for at least two weeks� is a rule of thumb to cover weekly cycles (e.g., weekday vs. weekend behavior). But the actual required duration is a function of three numbers:

• Daily traffic to the page or element being tested.
• Baseline conversion rate – lower baselines need larger samples.
• Minimum detectable effect (MDE) – the smallest improvement you care about. Smaller effects require dramatically more visitors.

For example, a high-traffic e-commerce site (100,000 visitors/day) testing a big button color change (MDE of 20%) might reach significance in 2-3 days. A low-traffic blog (500 visitors/day) testing a subtle headline change (MDE of 5%) could need several months. The calculator gives you the sample size; divide by your daily traffic to get the duration.

How to Choose Your Parameters

Selecting the inputs is a business decision, not just a math one. Here are practical guidelines:

• Baseline conversion rate: Use the last 4-6 weeks of historical data, not a single day. If you don’t have data, start with a reasonable estimate (e.g., 2-5% for e-commerce, 10-20% for email signups).
• Minimum detectable effect: Common choices are 5%, 10%, or 20% relative improvement. Smaller MDEs are tempting but require exponentially more traffic. If you can’t run long enough, increase the MDE.
• Power (1-β): 80% is standard. Higher power (90%) reduces the chance of missing a real effect but increases sample size by about 30%.
• Significance level (α): 5% is conventional. For high-risk changes (e.g., pricing), you might use 1% to reduce false positives.

If your estimated duration exceeds your patience or business cycle, you have three options: increase traffic (run on a more popular page), accept a larger MDE, or reduce power (not recommended below 70%).

Common Mistakes That Inflate Duration

• Peeking at results daily – stopping as soon as p < 0.05 inflates false positives. Decide the sample size before you start and stick to it.
• Ignoring weekly seasonality – if you run only 3 days, you may miss weekend behavior. Always include at least one full week.
• Using the wrong baseline – if you use a peak-season conversion rate, you’ll underestimate the required sample.
• Dividing sample size by total traffic instead of per-variant – the calculator output is per variant, so the total visitors needed is 2×n.
• Not accounting for existing traffic split – if you split 90/10 instead of 50/50, the smaller group needs more time.

Comparisons: Fixed Duration vs. Fixed Sample

There are two main approaches to planning an A/B test:

• Fixed sample size (recommended): Calculate n, run until each variant reaches n visitors, then analyze. This gives the most reliable results.
• Fixed duration: Choose a calendar period (e.g., 2 weeks), run, then check significance. This is simpler but risks either underpowered results (if traffic is lower than expected) or wasted time (if you could have stopped earlier).

Most testing tools default to a fixed-duration approach, but you can manually enforce a fixed sample by setting your test to pause when the target is reached. The calculator helps you set that target.

Safety and Validity Checks

Statistical validity depends on more than just sample size. Ensure:

• Random assignment of visitors to variants – no manual splitting or “first half of dayâ€� vs. “second halfâ€�.
• No changes to the page or other marketing campaigns during the test that could confound results.
• Use a consistent metric (e.g., conversion, revenue per visitor) and stick to it.
• Check that your tracking code is working correctly before the test starts.

If you violate these, even a perfectly sized test can give misleading conclusions.

Frequently Asked Questions

How many visitors do I need per day to run a two-week test?

Divide the sample size (n) from the calculator by 14. For example, if n = 31,196, you need about 2,228 visitors per day per variant, so roughly 4,456 total daily visitors to the page (assuming a 50/50 split). If your traffic is lower, you’ll need a longer test or a larger MDE.

Can I stop an A/B test early if the results look significant?

Only if you use a sequential testing method or a pre-planned interim analysis. Stopping as soon as p < 0.05 without adjusting for multiple looks dramatically increases your false positive rate. Use the calculator to set a fixed sample size and wait until it’s reached.

What if my baseline conversion rate changes during the test?

If there’s a known change (e.g., a marketing campaign), the test may be invalid. If it’s a gradual trend, consider using a longer baseline period or a time-series analysis. In practice, avoid launching other changes while the test runs.

Why does a 1% effect need so many more visitors than a 10% effect?

Because the formula divides by the square of the effect size. A 1% effect (0.01) squared is 0.0001, while a 10% effect (0.10) squared is 0.01 – a 100× difference. So the sample size increases by roughly the square of the inverse ratio. This is why small improvements are hard to detect.

Do I need to run the test for a full week even if I reach the sample size in 3 days?

Yes, if your traffic has weekly seasonality. Reaching the sample size in 3 days means you’ve only seen Monday-Wednesday behavior. Run at least 7 days to include a weekend, unless your page has no weekly pattern (e.g., a B2B tool used only on weekdays – then 5 days might suffice).

What’s the difference between statistical significance and practical significance?

Statistical significance means the result is unlikely due to chance; practical significance means the effect is large enough to matter to your business. A test can be statistically significant but show a 0.1% lift that isn’t worth the effort. The MDE you set in the calculator is your definition of practical significance.

Related

• 3D Touch Profile Picture Quick Contact Options
• Save Custom View: What It Means & How to Use It
• Is Easy Duplicate Finder Safe? File Deletion Explained

in base | Baseline conversion rate | % | 5 | 0
in mde | Minimum detectable effect | % | 10 | 0
in power | Statistical power | % | 80 | 0
in alpha | Significance level (alpha) | % | 5 | 0
out n | Sample size per variant | visitors | ceil((((z_alpha/2+z_power)^2)*(base*(1-base)+base*(1+effect)*(1-base*(1+effect))))/(effect^2)) | 0