Migrating to Kafka in Three Short Years

Three years ago, Etsy's analytics data pipeline was built around a pixel hosted on Akamai, FTP uploads, and Amazon EMR. Rafe Colburn, manager of the data engineering team at Etsy, talks about their migration to a data ingestion pipeline based on Kafka. He gives an overview on how they rebuilt their data pipeline without disrupting ongoing analytics work, as well as the tradeoffs made in building these systems.


This talk was given at the NYC Data Engineering meetup at Spotify.

This talk is included in our collection of the Top 12 Apache Kafka talks. Check out the others here.