In A/B testing, statistical significance means the difference between your variants is large enough, relative to the random noise in your data, that it would rarely appear if the change had no real effect. In practice you compute a p-value and call the result significant when it falls below a threshold you chose before the test, usually 0.05 (often described as "95% confidence"). Significance tells you a difference is probably not chance; it does not tell you the difference is large, valuable or permanent. Below is a complete worked example, a sample size table, and the mistakes that produce false winners.
What does statistical significance mean in A/B testing?
Every A/B test starts from a null hypothesis: the variant performs exactly the same as the control, and any gap you see is random variation in who happened to land in each group. Statistical significance is a decision rule for when the observed gap is too large to comfortably blame on that randomness. Five terms do all the work.
The five terms you need
- Significance level (alpha): the false positive rate you accept before the test. Alpha = 0.05 means that if the variant truly has no effect, you will still declare a winner about 5% of the time.
- Confidence level: 1 minus alpha. A 95% confidence level is the same rule as alpha = 0.05, just phrased from the other side. Testing tools that say "95% significance" usually mean this.
- p-value: the probability of seeing a difference at least as large as yours if the null hypothesis were true. If p is below alpha, the result is statistically significant.
- Statistical power: the probability that the test detects a real effect of a given size. The convention is 80%, meaning a 20% chance of missing a real effect (a false negative).
- Minimum detectable effect (MDE): the smallest lift you want the test to be able to detect reliably. It is a business decision, and it drives how much traffic you need.
Alpha, power, MDE and your baseline conversion rate together determine sample size. You fix all four before the test starts; the p-value is the only thing you calculate at the end.
How do you calculate A/B test significance? A worked example
Suppose you test a new checkout button. Traffic is split 50/50 and the test ran its planned length.
| Variant | Visitors | Conversions | Conversion rate |
|---|---|---|---|
| A (control) | 10,000 | 500 | 5.00% |
| B (variant) | 10,000 | 570 | 5.70% |
Is B's improvement statistically significant at alpha = 0.05, two-sided? The standard method for comparing two conversion rates is the two-proportion z-test.
Step 1: Conversion rates and lift
- pA = 500 / 10,000 = 0.0500
- pB = 570 / 10,000 = 0.0570
- Absolute difference = 0.0570 − 0.0500 = 0.0070 (0.70 percentage points)
- Relative lift = 0.0070 / 0.0500 = 14%
Step 2: Pooled conversion rate
Under the null hypothesis both variants share one true rate, so you pool the data: (500 + 570) / (10,000 + 10,000) = 1,070 / 20,000 = 0.0535.
Step 3: Standard error of the difference
SE = √[ p(1 − p) × (1/nA + 1/nB) ] = √[ 0.0535 × 0.9465 × (1/10,000 + 1/10,000) ] = √(0.0000101276) ≈ 0.00318.
Step 4: The z-score
z = (pB − pA) / SE = 0.0070 / 0.00318 ≈ 2.20. The observed difference is 2.2 standard errors away from zero.
Step 5: The p-value
For a two-sided test, p = 2 × (1 − Φ(2.20)), where Φ is the standard normal cumulative distribution. That gives p ≈ 0.028.
Step 6: The conclusion
0.028 is below 0.05, so the result is statistically significant at the 95% confidence level. In plain words: if the new button truly made no difference, a gap this large would show up in roughly 2.8% of tests this size. That is rare enough, by the rule you set in advance, to reject "no difference".
Step 7: A confidence interval for the difference
The p-value gives a yes or no; the confidence interval shows how big the effect plausibly is. For the interval you use the unpooled standard error:
SE = √[ pA(1 − pA)/nA + pB(1 − pB)/nB ] = √[ 0.05 × 0.95 / 10,000 + 0.057 × 0.943 / 10,000 ] ≈ 0.00318
95% CI = 0.0070 ± 1.96 × 0.00318 = 0.08 to 1.32 percentage points. Relative to the 5% baseline, that is a lift of roughly 1.5% to 26.5%.
This is the number to show stakeholders. The interval excludes zero (which is why the test is significant), but it is wide: the true lift could be almost nothing or more than a quarter. The 14% point estimate is the single most likely value, not a promise. Expect the effect you see after rollout to be smaller than 14% more often than larger, because winners selected for crossing a significance threshold tend to have overestimated effects.


What does 95% significance actually mean (and not mean)?
Misreading the 95% figure is the most common way marketers overstate test results.
| 95% significance means | 95% significance does NOT mean |
|---|---|
| If there were no real difference, a result this extreme would occur less than 5% of the time. | There is a 95% chance that B is better than A. |
| Across many tests with no real effect, you will falsely declare about 1 in 20 a winner. | There is only a 5% chance this particular winner is a false positive. |
| The observed difference is unlikely to be pure noise, under the test's assumptions. | The lift you measured is the lift you will get. |
| The decision rule held, provided you did not peek, change the setup or test many metrics. | The effect is large enough to matter to the business. |
"The probability B beats A" is a Bayesian quantity, and some tools report it. It answers a different question from a p-value; the trade-offs are covered in our guide to Bayesian vs frequentist A/B testing.
True or false: statistical significance is an optional part of A/B testing?
False. Without a significance test (or an equivalent Bayesian decision rule), an A/B test is just two numbers side by side, and two numbers will almost never be equal even when the variants are identical. If you ran the same page against itself, one copy would "win" every time. Significance testing is what separates a real effect from that guaranteed random gap, and it fixes your false positive rate in advance so you know how often you are wrong.
Skipping it has a predictable cost. Teams ship changes that did nothing, credit them with lifts that never show up in revenue, and build the next test on a false conclusion. What is genuinely optional is the specific threshold: 95% is a convention, not a law. A low-risk copy change might justify 90%; a pricing change might justify 99%. Choose the threshold before the test, write it in the plan, and do not move it after you see the data.
How much traffic does an A/B test need for significance?
The table below shows the required sample size per variant, computed with the standard two-proportion formula at 80% power and alpha = 0.05, two-sided:
n = [ z(1−α/2) √(2p̄(1 − p̄)) + z(1−β) √(p1(1 − p1) + p2(1 − p2)) ]² / (p2 − p1)²
where p1 is the baseline rate, p2 = p1 × (1 + MDE), p̄ is their average, z(1−α/2) = 1.96 and z(1−β) = 0.84. MDE is relative.
| Baseline conversion rate | MDE 5% relative | MDE 10% relative | MDE 20% relative |
|---|---|---|---|
| 1% | 637,010 | 163,095 | 42,693 |
| 2% | 315,206 | 80,682 | 21,109 |
| 3% | 207,938 | 53,211 | 13,914 |
| 5% | 122,124 | 31,234 | 8,158 |
| 10% | 57,763 | 14,751 | 3,841 |
Two patterns stand out. Halving the MDE roughly quadruples the traffic you need, and a lower baseline rate needs far more visitors. A store converting at 2% that wants to detect a 10% lift needs about 80,700 visitors per variant, or 161,400 in total. Different calculators use slightly different formulas, so expect small differences from these figures. Evan Miller's free sample size calculator is a good cross-check, and our A/B test sample size and duration guide covers turning these numbers into a test length.





