CRO
A/B Testing Statistical Significance: A Worked Example
EyeCaptain

A/B Testing Statistical Significance: A Worked Example

AB testing statistical significance explained with a worked z-test, p-value and confidence interval, a sample size table, and 6 mistakes that fake winners.

11 min read
398 views
statistical significance A/B testsample size calculationBayesian vs FrequentistA/B testing methodology

In A/B testing, statistical significance means the difference between your variants is large enough, relative to the random noise in your data, that it would rarely appear if the change had no real effect. In practice you compute a p-value and call the result significant when it falls below a threshold you chose before the test, usually 0.05 (often described as "95% confidence"). Significance tells you a difference is probably not chance; it does not tell you the difference is large, valuable or permanent. Below is a complete worked example, a sample size table, and the mistakes that produce false winners.

What does statistical significance mean in A/B testing?

Every A/B test starts from a null hypothesis: the variant performs exactly the same as the control, and any gap you see is random variation in who happened to land in each group. Statistical significance is a decision rule for when the observed gap is too large to comfortably blame on that randomness. Five terms do all the work.

The five terms you need

  • Significance level (alpha): the false positive rate you accept before the test. Alpha = 0.05 means that if the variant truly has no effect, you will still declare a winner about 5% of the time.
  • Confidence level: 1 minus alpha. A 95% confidence level is the same rule as alpha = 0.05, just phrased from the other side. Testing tools that say "95% significance" usually mean this.
  • p-value: the probability of seeing a difference at least as large as yours if the null hypothesis were true. If p is below alpha, the result is statistically significant.
  • Statistical power: the probability that the test detects a real effect of a given size. The convention is 80%, meaning a 20% chance of missing a real effect (a false negative).
  • Minimum detectable effect (MDE): the smallest lift you want the test to be able to detect reliably. It is a business decision, and it drives how much traffic you need.

Alpha, power, MDE and your baseline conversion rate together determine sample size. You fix all four before the test starts; the p-value is the only thing you calculate at the end.

How do you calculate A/B test significance? A worked example

Suppose you test a new checkout button. Traffic is split 50/50 and the test ran its planned length.

VariantVisitorsConversionsConversion rate
A (control)10,0005005.00%
B (variant)10,0005705.70%

Is B's improvement statistically significant at alpha = 0.05, two-sided? The standard method for comparing two conversion rates is the two-proportion z-test.

Step 1: Conversion rates and lift

  • pA = 500 / 10,000 = 0.0500
  • pB = 570 / 10,000 = 0.0570
  • Absolute difference = 0.0570 − 0.0500 = 0.0070 (0.70 percentage points)
  • Relative lift = 0.0070 / 0.0500 = 14%

Step 2: Pooled conversion rate

Under the null hypothesis both variants share one true rate, so you pool the data: (500 + 570) / (10,000 + 10,000) = 1,070 / 20,000 = 0.0535.

Step 3: Standard error of the difference

SE = √[ p(1 − p) × (1/nA + 1/nB) ] = √[ 0.0535 × 0.9465 × (1/10,000 + 1/10,000) ] = √(0.0000101276) ≈ 0.00318.

Step 4: The z-score

z = (pB − pA) / SE = 0.0070 / 0.00318 ≈ 2.20. The observed difference is 2.2 standard errors away from zero.

Step 5: The p-value

For a two-sided test, p = 2 × (1 − Φ(2.20)), where Φ is the standard normal cumulative distribution. That gives p ≈ 0.028.

Step 6: The conclusion

0.028 is below 0.05, so the result is statistically significant at the 95% confidence level. In plain words: if the new button truly made no difference, a gap this large would show up in roughly 2.8% of tests this size. That is rare enough, by the rule you set in advance, to reject "no difference".

Step 7: A confidence interval for the difference

The p-value gives a yes or no; the confidence interval shows how big the effect plausibly is. For the interval you use the unpooled standard error:

SE = √[ pA(1 − pA)/nA + pB(1 − pB)/nB ] = √[ 0.05 × 0.95 / 10,000 + 0.057 × 0.943 / 10,000 ] ≈ 0.00318

95% CI = 0.0070 ± 1.96 × 0.00318 = 0.08 to 1.32 percentage points. Relative to the 5% baseline, that is a lift of roughly 1.5% to 26.5%.

This is the number to show stakeholders. The interval excludes zero (which is why the test is significant), but it is wide: the true lift could be almost nothing or more than a quarter. The 14% point estimate is the single most likely value, not a promise. Expect the effect you see after rollout to be smaller than 14% more often than larger, because winners selected for crossing a significance threshold tend to have overestimated effects.

A/B Testing Reality Check: Why 90% of Your Tests Are Statistically Invalid infographic showing statistical significance A/B…EyeCaptain

What does 95% significance actually mean (and not mean)?

Misreading the 95% figure is the most common way marketers overstate test results.

95% significance means95% significance does NOT mean
If there were no real difference, a result this extreme would occur less than 5% of the time.There is a 95% chance that B is better than A.
Across many tests with no real effect, you will falsely declare about 1 in 20 a winner.There is only a 5% chance this particular winner is a false positive.
The observed difference is unlikely to be pure noise, under the test's assumptions.The lift you measured is the lift you will get.
The decision rule held, provided you did not peek, change the setup or test many metrics.The effect is large enough to matter to the business.

"The probability B beats A" is a Bayesian quantity, and some tools report it. It answers a different question from a p-value; the trade-offs are covered in our guide to Bayesian vs frequentist A/B testing.

True or false: statistical significance is an optional part of A/B testing?

False. Without a significance test (or an equivalent Bayesian decision rule), an A/B test is just two numbers side by side, and two numbers will almost never be equal even when the variants are identical. If you ran the same page against itself, one copy would "win" every time. Significance testing is what separates a real effect from that guaranteed random gap, and it fixes your false positive rate in advance so you know how often you are wrong.

Skipping it has a predictable cost. Teams ship changes that did nothing, credit them with lifts that never show up in revenue, and build the next test on a false conclusion. What is genuinely optional is the specific threshold: 95% is a convention, not a law. A low-risk copy change might justify 90%; a pricing change might justify 99%. Choose the threshold before the test, write it in the plan, and do not move it after you see the data.

How much traffic does an A/B test need for significance?

The table below shows the required sample size per variant, computed with the standard two-proportion formula at 80% power and alpha = 0.05, two-sided:

n = [ z(1−α/2) √(2p̄(1 − p̄)) + z(1−β) √(p1(1 − p1) + p2(1 − p2)) ]² / (p2 − p1)²

where p1 is the baseline rate, p2 = p1 × (1 + MDE), p̄ is their average, z(1−α/2) = 1.96 and z(1−β) = 0.84. MDE is relative.

Baseline conversion rateMDE 5% relativeMDE 10% relativeMDE 20% relative
1%637,010163,09542,693
2%315,20680,68221,109
3%207,93853,21113,914
5%122,12431,2348,158
10%57,76314,7513,841

Two patterns stand out. Halving the MDE roughly quadruples the traffic you need, and a lower baseline rate needs far more visitors. A store converting at 2% that wants to detect a 10% lift needs about 80,700 visitors per variant, or 161,400 in total. Different calculators use slightly different formulas, so expect small differences from these figures. Evan Miller's free sample size calculator is a good cross-check, and our A/B test sample size and duration guide covers turning these numbers into a test length.

EyeCaptainEyeCaptain
Conversions Booster

Your visitors leave without converting

Most websites lose 97% of their traffic without a single conversion. Our AI scans 200+ CRO elements to find exactly where visitors drop off.

Hero CTA missing

Look back at the worked example: 10,000 visitors per variant at a 5% baseline is only enough to reliably detect relative lifts of about 18% or more. The test reached significance, but with that little traffic, a significant result is more likely than a well-powered one to overstate the true effect.

A/B Testing Reality Check: Why 90% of Your Tests Are Statistically Invalid infographic showing statistical significance A/B…EyeCaptain

Common A/B test significance mistakes

1. Peeking and stopping early

The p-value is valid only if you look once, at the planned sample size. Checking repeatedly and stopping the first time p dips below 0.05 inflates false positives sharply. We simulated 20,000 A/A tests (identical variants, 5% conversion rate, 10,000 visitors each) and checked each one at 10 evenly spaced points. Testing only at the end produced false winners 4.9% of the time, as designed. Stopping at the first significant look produced false winners 19% of the time, nearly four times the stated rate. Evan Miller makes the same point in "How Not To Run an A/B Test". If you need to monitor tests, use a method designed for it, such as sequential testing. See the peeking problem explained for the options.

2. Underpowered tests

A test without enough traffic for its MDE will usually come back inconclusive, and the occasional significant result will tend to exaggerate the effect. If the table says you need 80,000 visitors per variant and you get 10,000 a month, test bolder changes (a larger MDE) or a higher-traffic page, rather than running a test that cannot answer the question. If you are already stuck with a flat result, read what to do when an A/B test is inconclusive.

3. Multiple comparisons: many variants, many metrics

Every additional comparison is another chance at a false positive. If none of your changes do anything, the probability of at least one false "win" at alpha = 0.05 is 1 − 0.95^k for k independent comparisons:

  • 4 variants against one control: 1 − 0.95^4 ≈ 18.5%
  • 20 metrics on one test: 1 − 0.95^20 ≈ 64%

Pick one primary metric before the test and judge the test on it alone. Treat secondary metrics as diagnostics. If you must compare several variants, apply a correction such as Bonferroni (divide alpha by the number of comparisons: 0.05 / 4 = 0.0125) and size the test for it.

4. Sample ratio mismatch (SRM)

If you split traffic 50/50 but end up with 10,000 visitors in A and 9,500 in B, something is broken: a redirect that loses users, a bot filter applied to one arm, or a tracking script that fires late on the variant. Kohavi, Tang and Xu devote a chapter of Trustworthy Online Controlled Experiments (Cambridge University Press, 2020) to sample ratio mismatch as a trust check. Test it with a chi-square goodness-of-fit test on visitor counts. For 10,000 vs 9,500 against an expected 9,750 each, chi-square = (250² + 250²) / 9,750 ≈ 12.8, p ≈ 0.0003. That split is almost certainly not chance, and the conversion results cannot be trusted until the cause is found.

5. Novelty effects and partial business cycles

Returning visitors may click a new design simply because it is new, and the effect fades. Behaviour also differs by weekday, payday and campaign calendar. Run every test for at least one full week, ideally two, and in whole-week blocks even if you hit your sample size sooner. A test that ran Monday to Thursday has measured Monday-to-Thursday visitors only.

6. Stopping on statistical significance alone

Statistical significance is not practical significance. With enough traffic, a 0.5% relative lift becomes significant, but it may not cover the cost of building and maintaining the change. Before the test, decide the smallest lift worth shipping, and after the test check whether the confidence interval clears it. In the worked example the interval runs from about 1.5% to 26.5% relative: significant, but it does not rule out a lift too small to matter.

A/B Testing Reality Check: Why 90% of Your Tests Are Statistically Invalid infographic showing statistical significance A/B…EyeCaptain

A/B test statistics: pre-test checklist

  1. Write a hypothesis that names the change, the metric and the reason. Our guide on how to write an A/B test hypothesis has a template.
  2. Choose one primary metric and list secondary metrics as diagnostics only.
  3. Record the baseline conversion rate from at least two recent weeks.
  4. Set the MDE you care about, alpha (usually 0.05, two-sided) and power (usually 80%).
  5. Calculate the sample size per variant and convert it to a duration in whole weeks.
  6. Decide the analysis method (fixed horizon or sequential) and do not change it later.
  7. Run an A/A test or check the split on day one to catch sample ratio mismatch early.
  8. Define the smallest lift worth shipping and what you will do if the result is inconclusive.

Traffic is the scarce resource, so spend it on the changes most likely to matter. A prioritisation model like ICE, PIE or PXL helps you rank ideas. EyeCaptain is not an A/B testing tool, but a free EyeCaptain page audit gives you a list of issues ranked by impact, which can feed that backlog before you commit weeks of traffic to a test. For the full testing process from idea to rollout, see our A/B testing guide.

Frequently asked questions

What is a good p-value for an A/B test?

The common threshold is 0.05, which corresponds to 95% confidence. There is nothing special about 0.05; it is a convention that trades off false positives against the traffic you need. Choose the threshold before the test starts based on the cost of a wrong decision, and never adjust it after you see the results.

Is 90% confidence enough for an A/B test?

It can be, for low-risk changes that are cheap to reverse. At 90% confidence you accept a 10% false positive rate when there is no real effect, twice the rate at 95%. If you use 90%, decide that in advance and size the test for it rather than lowering the bar because a test fell short of 95%.

What is the difference between statistical significance and confidence level?

The confidence level is 1 minus the significance level (alpha). A 95% confidence level and a 0.05 significance level describe the same decision rule. A result is statistically significant when its p-value falls below alpha.

Can an A/B test be significant but wrong?

Yes. At alpha = 0.05, about 1 in 20 tests of changes with no real effect will still come out significant. Peeking, testing many metrics or variants, and sample ratio mismatch push that rate much higher, which is why the analysis plan must be fixed before the test starts.

How long should an A/B test run to reach significance?

Run it until it reaches the sample size you calculated in advance, rounded up to whole weeks, and for at least one full week. Do not stop the moment the tool shows significance. If the required sample size would take months, test a bolder change or a page with more traffic instead.

Enjoyed this article?

One short email a week with practical CRO and UX tips

No spam. Unsubscribe anytime.

Find what stops your visitors from converting

EyeCaptain is an AI-powered CRO & UX audit tool that automatically scans your pages, identifies UX issues, and gives you actionable optimization suggestions to increase conversions. Try it for free, no card, no commitment.

Free Website Audit
Share this article
LinkedIn X

About the author

Founder of EyeCaptain. Writes about conversion rate optimization, UX analysis and the neuromarketing behind pages that actually sell.

More articles by Dimitris Andreadakis