Meetup Audio - NYC Machine Learning: Experimentation & Visualization Honing Machine Learning on Big Data

In this audio recording from at the NYC Machine Learning Meetup, Jeremy Stanley (SVP, Product and Data Sciences at Collective) talks about Using Pervasive Experimentation & Visualization to Hone Machine Learning on Big Data

Abstract (from speaker)

How do you know if your machine learning system works? You've tested it on a held out set of data, but is that enough? What if you have 1,000s of models in production influencing billions of decisions in a chaotic, real time environment? What if you have to convince other internal departments critical to implementation success, your CEO, your customers?

At Collective, we have found that a combination of continuous 'live' experimentation, coupled with copious visualization have been key to our successes with machine learning in practice. This talk will cover our Audience Optimization system, which impacts billions of advertising impressions monthly. We continuously optimize thousands of campaigns over hundreds of millions of users, in multiple continents, against varying performance objectives.

Specifically, the talk will focus on:


  • The problems Audience Optimization addresses

  • How the system is used in production

  • The process and supporting architecture

  • The specific algorithms used

  • How we experiment with & visualize (almost) everything, always

  • How we prove our system works (to ourselves and our clients)

  • How we glean insights from visualizations to improve the system


The majority of the time will be spent on the final 3 points. This presentation should be of interest to any practitioners who are seeking to implement and continuously improve a large scale production machine learning system on big data.

Want to hear from more top engineers?

Our weekly email contains the best software development content and interviews with top CTOs. Enter your email address now to stay in the loop.