Experimentation and Interference in a Two-Sided Marketplace
Simple “random-user” A/B experiment designs fall short in the face of complex dependence structures. These can come in the form of large-scale social graphs or, more recently, spatio-temporal network interactions in a two-sided transportation marketplace. Naive designs are susceptible to statistical interference, which can lead to biased estimates of the treatment effect under study.
In this talk we discuss the implications of interference for the design and analysis of live experiments at Lyft. A link is drawn between design choices and a spectrum of bias-variance tradeoffs. We also motivate the use of large-scale simulation for two purposes: as an efficient filter on candidate tests, and as a means of justifying the assumptions underlying our choice of experimental design.
Nick Chamandy is the Head of Data Science at Lyft. Before joining Lyft, he learned his trade as a member of the Ads Quality Team at Google. Nick got his PhD in Statistics from McGill University, where he studied brain imaging applications under the supervision of Keith Worsley and Russ Steele. Currently, Nick's work centers around travel time prediction, dynamic pricing, and experimentation. He is fascinated by new data science problems, and driven by the prospect of building better products, and sharing deeper insights, with data.
This talk was given at the SF Data Engineering meetup in May 2016.