Real-time Streaming and Data Pipelines with Apache Kafka

In this talk, Joe Stein, Apache Kafka committer, member of the PMC, and Founder and Principal Architect at Big Data Open Source Security, will talk on Apache Kafka an open source, distributed publish-subscribe messaging system. Joe will focus on how to get started with Apache Kafka, how replication works and more! Storm is a great system for real-time analytics and stream processing but to get the data into Storm, you need to collect your data streams with consistency and availability at high loads and large volumes. Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. This talk was recorded at the NYC Storm User Group meetup at WebMD Health.

58:26

More info: Apache Kafka (http://kafka.apache.org/is fast, a single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It's also scalable, Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers. It can also be durable, messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact. Plus it's Distributed by Design, Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.

We have many more articles on Apache Kafka. Check out our collection of top 12 tech talks on Apache Kafka.

 Bio: Joe Stein is an Apache Kafka committer and member of the PMC and is the Founder and Principal Architect at Big Data Open Source Security LLC http://www.stealth.ly

Want to hear from more top engineers?

Our weekly email contains the best software development content and interviews with top CTOs. Enter your email address now to stay in the loop.