Spotify Tech Lead Kinshuk Mishra and Engineer Noel Cody share their experience about building personalized ad experiences for users through iterative engineering and product development. They explain their process of continuous problem discovery, hypothesis generation, product development and experimentation. Later they deep dive into the specific ad personalization problems Spotify is solving and explain their data infrastructure technology stack in detail. They also speak about how they've experimented various product hypothesis and iteratively evolved their infrastructure to keep up with the product requirements.
Understanding the billions of data points we ingest each month is no easy task. Through the development of models that allow us to do so, we’ve noticed some commonalities in the process of converting raw data to real-world understanding. Although you can get pretty good results with simple models and algorithms, digging beyond the obvious abstractions and using more sophisticated methods requires a lot of effort. In school we often learn different techniques and algorithms in isolation, with neatly fitted input sets, and study their properties. In the real world, however, especially the world of location data, we often need to combine these approaches in novel ways in order to yield usable results.
In this article we look at the process of understanding the importance of different locations as they relate to consumers, which takes us from simple joins in Hadoop to a sophisticated time-series algorithm called Viterbi.
As online advertising has grown from an experiment on a marketer’s checklist to a critical tool in the proverbial toolbox, so has the demand for actionable metrics of performance.
At first, measuring engagement was straightforward. A site serves a user an ad (delivered by an unbiased third-party, the ad server), and a user clicks on that ad to go to whatever page the marketer desired. Ad servers then collect the number of clicks and impressions, which serves two primary purposes. The first is that marketers use these numbers to draw insights into how their campaigns are performing. The second is that marketers pay their advertising partners based on things like number of clicks.
Soon, marketers clamored to gain deeper insights. Technology vendors introduced cookies to attribute actions on the site, such as a product purchase or online signup, called a “conversion,” to an ad impression or click. It’s this process — attributing actions on a site to ad impressions and clicks — where things get tricky, and which this blog post will attempt to explain.
Tapad Director of Engineering, Toby Matejovsky talks about how his team built and scaled their cross device digital advertising platform that handles over 50,000 queries per second per server with sub-millisecond latency, 95-99% of the time. Toby shares lessons learned, scaling tips and best practices as well as answer questions ranging from tools and technologies to people and processes.
Here's a new talk on targeted online advertising recorded at one of the NYC Machine Learning meetups. Two presenters from Media6 labs spoke about their respective papers from the recent Knowledge Discover and Data Mining conference (KDD). Claudia Perlich presented "Bid Optimizing and Inventory Scoring in Targeted Online Advertising" and Troy Raeder presented "Design Principles of Massive, Robust Prediction Systems." Full abstracts and audio below.
Description from Meetup.com:
At the Machine Learning Meetup hosted at Pivotal Labs, two presenters from Media6 labs each talked about their papers from Knowledge Discover and Data Mining (KDD). Claudia Perlich presented"Bid Optimizing and Inventory Scoring in Targeted Online Advertising" and Troy Raeder presented "Design Principles of Massive, Robust Prediction Systems". Abstracts and bios below.
Bid Optimizing and Inventory Scoring in Targeted Online Advertising
Billions of online display advertising spots are purchased on a daily basis through real time bidding exchanges (RTBs). Advertising companies bid for these spots on behalf of a company or brand in order to purchase these spots to display banner advertisements. These bidding decisions must be made in fractions of a second after the potential purchaser is informed of what location (Internet site) has a spot available and who would see the advertisement. The entire transaction must be completed in near real-time to avoid delays loading the page and maintain a good users experience. This paper presents a bid-optimization approach that is implemented in production at Media6Degrees for bidding on these advertising opportunities at an appropriate price. The approach combines several supervised learning algorithms, as well as second price auction theory, to determine the correct price to ensure that the right message is delivered to the right person, at the right time.
Design Principles of Massive, Robust Prediction Systems
Most data mining research is concerned with building high-quality classification models in isolation. In massive production systems, however, the ability to monitor and maintain performance over time while growing in size and scope is equally important. Many external factors may degrade classification performance including changes in data distribution, noise or bias in the source data, and the evolution of the system itself. A well-functioning system must gracefully handle all of these. This paper lays out a set of design principles for large-scale autonomous data mining systems and then demonstrates our application of these principles within the m6d automated ad targeting system. We demonstrate a comprehensive set of quality control processes that allow us monitor and maintain thousands of distinct classification models automatically, and to add new models, take on new data, and correct poorly-performing models without manual intervention or system disruption.
Troy Raeder Bio:
Troy Raeder has earned B.S., M.S., and Ph.D degrees in Computer Science from the University of Notre Dame and is currently a Data Scientist at M6D. He has published academic articles in a number of venues including Pattern Recognition and the Journal of Machine Learning Research. His current research interests include machine learning for online advertising, learning under shifting distributions, and the development large-scale machine learning algorithms and systems.
Claudia Perlich Bio:
Since 2010, Claudia Perlich holds the position of chief scientist at Media6Degrees, a startup that specializes at targeted online display advertising. Claudia received her Ph.D. in Information Systems from Stern School of Business, New York University in 2005 and holds additional graduate degrees in Computer Science. Claudia joined the Data Analytics Research group at the IBM T.J. Watson Research Center in 2004 and continued her research on data analytics and machine learning for complex real-world domains and applications. She is the author or 50+ scientific publications and holds multiple patents in the area of machine learning, has won various data mining competitions, best paper awards, and speaks regularly at conferences and other public events.
Dag Liodden, VP of Engineering at Tapad, an NYC startup in the mobile advertising space, talks with me about what Tapad does, the high-scale requirements of their business, and why they moved off the cloud.