Bitly - Realtime Distributed Message Processing at Scale with NSQ

This talk is by Jehiah Czebotar, Lead Engineer at Bitly, and Matt Reiferson, a Software Engineer at Bitly, recorded at the eBay NYC offices. This talk was recorded at the NYC Data Engineering meetup at Ebay NYC.

Matt and Jehiah will be talking about NSQ, their open sourced project  that solves an issue of  realtime distributed message processing, designed to operate at bitly’s scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.



Bios: Matt Reiferson is a software engineer, lunch instructor, and fan of tv's Bob Ross. Matt is an Engineer at bitly, one of the main architects behind NSQ. He is most known as the creator and host of The Joy of Coding in DOS, a television program that ran for twelve years on PBS stations in the United States.

Jehiah is a co-author of NSQ. When not painstakingly shortening urls by hand, Jehiah can be found riding trains, learning Amharic, or building iPhone apps to help others do both of those things more efficiently. A lover of data, he has been known to compile amazing personal annual reports documenting the details of his day to day life that most people take for granted, which you can see here:

Want to hear from more top engineers?

Our weekly email contains the best software development content and interviews with top CTOs. Enter your email address now to stay in the loop.

[caption id="attachment_784" align="alignleft" width="550"]Moment of the presentation Moment of the presentation[/caption]

Some background on NSQ

NSQ - realtime distributed message processing at scale. Let's examine what that means. NSQ is used in the backend of software architectures to process large quantities of messages. The scale NSQ is designed to handle is in the order of billions of messages per day or even per hour. It is designed to do that without a single point of failure which means that even at incredible scale and load, it is still extremely reliable. NSQ does not depend on data structure of a database. It can process data in JSON forma or a number of other protocols.

And right from the Bitle website, here is what NSQ is designed to do:

  • provide easy topology solutions that enable high-availability and eliminate SPOFs

  • address the need for stronger message delivery guarantees

  • bound the memory footprint of a single process (by persisting some messages to disk)

  • greatly simplify configuration requirements for producers and consumers

  • provide a straightforward upgrade path

  • improve efficiency

Want to get updates about NSQ and other open source technologies?

If you'd like to get updates about NSQ and other open source technology developments, especially in the world of big data, you are welcome to subscribe to our software engineering and open source technology newsletter. Or you can follow the founder of g33ktalk Pete Soderling on Twitter @petesoder.

Additionally, we hold regular events on software engineering, open source technology, big data and other topics at our San Francsico and New York engineering meetups.

Thanks to NSQ authors

NSQ was designed and developed by Matt Reiferson (follow on Twitter @imsnakes) and Jehiah Czebotar (follow on Twitter @jehiah). And Bit.y was the company who contributed and supported NSQ.