Matched Pair A/B Testing for LLM Prompts
The calculator for this page is switched off while it is checked. The formula is below, and the worked example in the article shows how to use it by hand.
Matched pair A/B testing for LLM prompts is a method where you compare two prompt versions against the same set of inputs, using paired statistical tests to determine which performs better. You enter your two prompts and a set of test inputs, and the calculator applies a paired t-test to the metric scores you provide, returning a p-value and confidence interval. This approach controls for input variability, making it far more sensitive than comparing averages across different random samples.
Formula
The paired t-test compares the mean of the differences (d = score_A – score_B) against zero. The test statistic is:
t = (mean(d)) / (sd(d) / sqrt(n))
where:
- d = difference between Prompt A and Prompt B score for each input (unit: score points)
- mean(d) = average of all differences (score points)
- sd(d) = sample standard deviation of the differences (score points)
- n = number of paired inputs (count)
- t = t-statistic with n-1 degrees of freedom
The p-value is the probability of observing a t-statistic this extreme if the true mean difference is zero. A p-value below your chosen significance level (commonly 0.05) suggests the difference is not due to random chance.
Worked Example
Using the default values: Prompt A scores = [8, 7, 9, 6, 8], Prompt B scores = [6, 5, 7, 4, 6].
- Compute differences: d = [2, 2, 2, 2, 2]
- Mean difference = (2+2+2+2+2)/5 = 2.0
- Standard deviation of differences = 0 (all differences are 2)
- t = 2.0 / (0 / sqrt(5)) = undefined (division by zero)
Since the standard deviation is zero, the t-statistic cannot be computed. This is a real edge case: if every input produces the same difference, the test is degenerate. In practice, you would not need a test—the result is consistent. The calculator returns an error or a p-value of 1, depending on implementation. For a meaningful example, use scores with some variation, such as A = [8, 7, 9, 6, 8] and B = [7, 5, 8, 4, 6], giving differences [1, 2, 1, 2, 2], mean = 1.6, sd = 0.548, t = 1.6 / (0.548/sqrt(5)) = 6.53, p ≈ 0.003. That would be a statistically significant improvement.
Why Use Matched Pairs for LLM Prompts?
LLM outputs are highly sensitive to input wording, topic, and even random seed. If you test Prompt A on one set of questions and Prompt B on another, differences in difficulty or ambiguity can swamp the actual prompt effect. Matched pairs eliminate this by feeding the exact same inputs to both prompts. This is especially important when your metric is subjective (e.g., helpfulness, tone) or when your test set is small. For example, if you have 20 customer queries, you can run both prompts on all 20 and compare the scores per query. This design also allows you to see which inputs cause the biggest divergence, guiding further prompt iterations.
How to Choose a Metric
The paired test only works if you have a numeric score for each output. Common choices include:
- Accuracy: binary (correct/incorrect) or partial credit (0-1)
- Rouge-L / BLEU: for summarization or translation tasks
- LLM-as-judge: a separate LLM rates outputs on a 1-5 scale
- Human rating: average of multiple raters per output
- Latency or cost: if that is your optimization target
Whatever you choose, it must be applied consistently to both prompts. For binary outcomes, a paired design is equivalent to McNemar’s test, which you can also compute from the discordant pairs. For ordinal scales, you could use a Wilcoxon signed-rank test, but the paired t-test is usually robust enough with sample sizes above 10.
Common Pitfalls and Misconceptions
A frequent mistake is treating the paired test as a silver bullet when the metric itself is noisy. If your LLM-judge has high variance, even a paired design may not detect a real difference. Another pitfall is ignoring the order of presentation: if you always run Prompt A first, the model’s context or randomness might be affected. Randomize the order per input. Also, beware of multiple comparisons: if you test 10 prompt variations on the same data, the chance of a false positive increases. Use a correction like Bonferroni or simply treat early tests as exploratory. A common misconception is that a p-value above 0.05 means the prompts are equivalent. It only means the data are insufficient to prove a difference. You might need more samples or a better metric.
Sample Size and Statistical Power
How many paired inputs do you need? It depends on the effect size you expect. A rough rule: for a small effect (d = 0.2), you need about 200 pairs for 80% power at alpha = 0.05. For a large effect (d = 0.8), about 15 pairs suffice. In practice, with LLM prompts, effects are often moderate to large, so 20-50 inputs are a common starting point. But if your metric is noisy (e.g., a 1-5 scale with high variance), you may need more. You can compute power beforehand using a paired t-test power calculator, or simply collect data in batches and check the confidence interval width. If the interval is too wide to be useful, add more inputs.
Alternatives to Matched Pairs
If you cannot reuse the same inputs (e.g., prompts are designed for different tasks), you might use an independent t-test, but that requires larger sample sizes and is more vulnerable to confounding. Another alternative is a crossover design where each input is used with each prompt but on different days, which is a form of pairing. For non-numeric outcomes, you might use a chi-square test on counts. For prompt optimization, you might also consider Bayesian approaches that model the probability that Prompt A is better, which can be more intuitive than p-values. However, matched pairs remain the simplest and most robust for most LLM prompt comparisons.
Frequently Asked Questions
What if my metric is not normally distributed?
The paired t-test assumes the differences are approximately normal. With small samples, this can be an issue. You can use the Wilcoxon signed-rank test as a non-parametric alternative, which only assumes symmetry of differences. In practice, the t-test is fairly robust for sample sizes above 20, but if you have many outliers, consider the Wilcoxon test.
Can I use this for comparing more than two prompts?
Yes, but you should use a repeated-measures ANOVA or Friedman test to avoid inflating the error rate. If you simply run multiple paired t-tests, apply a correction like Bonferroni. The calculator here is for two prompts; for more, you would need a different tool.
What if my scores are percentages or ratios?
You can still use the paired t-test as long as the differences are meaningful. However, if scores are bounded (e.g., 0-100%), the differences might be heteroscedastic. In that case, you might transform the data (e.g., logit) or use a generalized linear mixed model. For most practical purposes, the t-test on raw differences is acceptable.
How do I handle missing data?
If one prompt fails to produce an output for a given input, you typically exclude that pair from the analysis. If you have many missing values, consider whether the failure is related to the prompt—that itself is a signal. You might then use a different metric that accounts for failures, or analyze the failure rate separately.
Is a p-value of 0.05 always the right threshold?
No. The threshold depends on the cost of a false positive. If you are choosing between two prompts for a high-stakes application, you might want a stricter threshold like 0.01. If you are just exploring, 0.1 might be fine. Always report the p-value and confidence interval, not just a pass/fail.
Related
Formula used
in a | Prompt A scores | - | 8, 7, 9, 6, 8 | 0 in b | Prompt B scores | - | 6, 5, 7, 4, 6 | 0 out p | p-value (two-tailed) | - | tdist(t, n-1) | 4 out t | t-statistic | - | (mean_diff) / (sd_diff / sqrt(n)) | 3 out ci | 95% CI of mean diff | - | mean_diff ± tcrit * (sd_diff / sqrt(n)) | 3
Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.