Storm at Spotify: Deployment and Use Cases

This is the first time that a Spotify engineer has spoken publicly about their deployment and use cases for Storm! In this talk, Software Engineer Neville Li describes:


  • Real-time features developed using Storm and Kafka including¬†recommendations, social features, data visualization and ad targeting

  • Architecture

  • Production integration

  • Best practices for deployment


49:54

Spotify is an exciting case study - users create 600 Gigabyte of data per day and 150 Gigabyte of data per day via different services. Every day 4 Terabyte of data is generated in Hadoop, a 700-node cluster running over 2.000 jobs per day. They currently have 28 Petabytes of storage, spread out over 4 data centres across the world.

See more Apache Kafka tutorials on our compilation article for best Apache Kafka tech talks.

This talk was presented at the New York City Storm User Group hosted at Spotify HQ in NYC.