This workshop offers an end-to-end introduction of A/B test planning and result analysis in Python. We will cover best practices for power calculations, post-experiment analysis, delving into packages available and techniques appropriate for different types of data.
Abstract
This workshop offers an introduction to the full lifecycle of an A/B test from a data science point of view – from planning an experiment and estimating the experiment duration, to analyzing the results by using the appropriate statistical techniques, and finally presenting the conclusions with compelling and meaningful visualizations.
During the hands-on workshop, we will work as data scientists tasked to support a product experiment from beginning to end. The steps we will cover are:
- Explore pre-experimental data (EDA)
- Choose the appropriate experiment design
- Estimate how long we should run the experiment for
- Setup monitoring (visualizations and statistics) and understand pitfalls of “peeking”
- Check results to ensure data was correctly generated.
- Analyze results using frequentist methods (selecting the correct approach for each type of metric)
- Present your findings in compelling visualization.
In the process, we will discuss the most popular libraries that can help us with (frequentist) A/B test analysis (e.g. pandas, statsmodels, scipy), as well as smaller libraries for specific applications (e.g. ssrm-test). If time permits, we can also take a look at alternative approaches, such as bootstrapping and Bayesian inference.
After attending the workshop, you should have the tools (and a few example notebooks!) available to further explore any A/B tests that might come your way.
