What is Experimentation and A/B Testing?
Experimentation in marketing means running controlled tests, most often an A/B test that randomly shows two versions of a page, email or ad to different people and compares the results.
Why does Experimentation and A/B Testing matter for marketers?
It is the most direct way to learn whether a change caused an improvement, which is why it is used to check other forms of measurement.
Where does Experimentation and A/B Testing stand today?
Large sites run experiments continuously: Wikipedia reports that Google ran more than 7,000 tests a year by 2011 and that a single headline test at Bing produced a 12% revenue increase in 2012. Vendors in this directory include Optimizely, whose company grew from an A/B testing product, and Amplitude, which sells web and feature experimentation.
Common use cases
- Testing headlines, layouts and calls to action
- Checking whether a new feature changes behavior
- Measuring the lift from a campaign
In an A/B test, people are divided at random between two versions and the outcomes compared. Wikipedia’s example is an email sent with different calls to action to 1,000 people each. The results then need statistical testing to show that the difference is not down to chance.
Limits
A/B tests are sensitive to variance and need large sample sizes to reduce standard error, and they can waste effort when the result is not useful. They answer narrow questions well; broader questions about channels are usually handled with attribution and marketing mix modelling.