Netflix: Controlled Experimentation to Guide Product Innovation by Caitlin Smallwood
Controlled Experimentation (or A/B testing) has evolved into a powerful tool for driving product strategy and innovation. The dramatic growth in online and mobile content, media, and commerce has enabled companies to make principled data-driven decisions. Large numbers of experiments are typically run to validate hypotheses, study causation, and optimize user experience, engagement, and monetization.
The concept of controlled experimentation is simple - randomly divide the user population into two groups called the Control (A) and Treatment (B). The experiment involves concurrently exposing the Control group users to one experience (typically, the existing experience) and the Treatment group users to another (the new experience). A set of performance metrics are computed for both groups and statistical tests are run to determine if the change in metrics (if any) for the Treatment group compared to the Control group is purely due to chance or not. Typical use cases of A/B testing include, testing a modified web user interface, evaluating a new call to action for mobile app downloads, and examining the effects of a new personalization algorithm.
Want to hear from more top engineers?
Our weekly email contains the best software development content and interviews with top CTOs. Enter your email address now to stay in the loop.