Mathbabe: A recommendation system and MapReduce by Cathy O’Neil logo

In this talk, we'll see how recommendation systems are created from data. What's the algorithm? What's the evaluation method? What's the optimization procedure? When does it converge? We'll talk about parallelizing in order to scale up to "big data" size via the MapReduce framework. Finally, we'll think about priors and how they are overloaded. Content from this talk draws from chapters in Doing Data Science contributed by David Crawshaw and Matt Gattis. This talk was recorded at the NYC Machine Learning meetup at Pivotal Labs.

Cathy O’Neil, PhD, Senior Researcher at Johnson Research Labs

Cathy O’Neil earned a Ph.D. in math from Harvard, was postdoc at the MIT math department, and a professor at Barnard College where she published a number of research papers in arithmetic algebraic geometry. She is co-authoring a book (with Rachel Schutt) called “Doing Data Science” to be published by O’Reilly in Spring 2013. She previously worked as a quant for the hedge fund D.E. Shaw in the middle of the credit crisis, and then for RiskMetrics, a risk software company that assesses risk for the holdings of hedge funds and banks. She facilitates an Occupy Wall Street working group, called Alternative Banking, and writes a blog at

Want to hear from more top engineers?

Our weekly email contains the best software development content and interviews with top CTOs. Enter your email address now to stay in the loop.