A/B Testing
Running an experiment that supports the conclusion you draw from it.
Questions
Easy / Med / Hard
Your accuracy
An A/B test is a randomised experiment, and most of what goes wrong is decided before any data arrives.
Randomisation is what makes the comparison valid. Assigning users at random means the groups differ only by chance, so a difference in outcome can be attributed to the treatment. Assigning by anything correlated with behaviour — signup date, region, device — destroys that and the test measures the assignment rule.
Fix the sample size in advance. A power calculation combines your baseline rate, the minimum effect worth detecting, the significance level, and the desired power. Running until you like the answer is not an experiment.
Peeking is the most common way to get a false positive. Checking significance repeatedly and stopping when p first drops below 0.05 dramatically inflates the false positive rate, because with enough looks a random walk crosses the threshold eventually. Either commit to the planned duration or use a method designed for sequential testing.
Multiple comparisons do the same thing across metrics. Test twenty metrics at 0.05 and you expect one false positive by construction. Declare the primary metric before you start; treat the rest as exploratory.
Statistical significance is not practical significance. With a large enough sample, a 0.01% improvement becomes significant and is still not worth shipping. Ask what effect size would actually change the decision.
Novelty and primacy effects distort early results in opposite directions: existing users click a new thing because it is new, or resist it because it is different. Both fade, which is why very short tests on established products mislead.
Interference breaks the independence assumption. In marketplaces and social products, treating one user affects others — a treated seller takes demand from a control seller — so the control group is no longer clean.
A sample ratio mismatch is a stop sign. If a 50/50 split arrives 52/48, something is wrong with assignment or logging, and the results should not be trusted until you know what.