Twitter Heron at Scale

Twitter generates billions and billions of events per day. Analyzing these events in real time presents a massive challenge. Twitter designed and deployed a new streaming system called Heron. Heron has been in production nearly 2 years and is widely used by several teams for diverse use cases. Twitter open-sourced Heron this year.

In this talk, you will learn about the operating experiences and challenges of running Heron at scale and the approaches that the team at Twitter took to solve those challenges.

Karthik Ramasamy is the engineering manager and technical lead for real time analytics at Twitter. He is the co-creator of Heron and has more than two decades of experience working in parallel databases, big data infrastructure and networking. He cofounded Locomatix, a company that specializes in real time streaming processing on Hadoop and Cassandra using SQL that was acquired by Twitter.

This talk is from DataEngConf SF in April 2016.