Step 1 of 11
Start here: is B's lead real?
This A/B testing lesson starts with a question every team meets: a new version of a page (B) is ahead of the old one (A) after a day. Do you switch? Many teams get it wrong, because chance alone can open a lead between two pages that are exactly as good.
Watch the visitors split between the two versions and the bars on the right. By the end of this lesson you'll know when B's bar has really pulled clear, and when it only looks that way.
What you'll do
- Split visitors at random, and see what goes wrong when the split isn't random
- Plan a test before it starts: the metric, what gets split, when it stops
- Run tests hundreds of times in a simulator, to see the gaps chance makes on its own
- Catch the two most common ways tests fool their teams: stopping early and checking too many numbers
By the end you can
- read a p-value in plain words
- work out how many visitors and days a test needs
- tell a real winner from a lucky one
Every test runs in the lesson: no statistics background and no software needed.