Normal bitly 1400x336


New York, NY

About Bitly Engineering

At Bitly, we are obsessed with making the internet work better. We started in 2008 as a humble URL shortener and have evolved into an real-time intelligence engine, fueled by billions of clicks and petabyes of data. Our comprehensive social media monitoring platform leverages our unique scale and empowers The New York Times, ESPN and IBM (to name a few) to monitor, understand, and engage with their audiences.

We're predominately a Python shop, but we don't stop there. Oh no, we use Ruby, Java, C, C++, Go, and even a little Tickle. We've got nerf guns, but that doesn't stop us from solving problems at scale without breaking a sweat. Fine, sometimes we sweat, but we still look good.
You'll be working in a flat, transparent, fast-paced start-up environment in the heart of Union Square NYC. Most importantly, we offer you the chance to work with smart people on difficult problems at huge scale that impact how millions of people use the Internet.

Small sean o connor Sean O'Connor on

Bitly's Practical Strategies for Building Distributed Systems

Over the years, bitly has built a number of large scale systems to handle and analyze billions of clicks each month. Distributed systems can often be challenging to build and operate, but they can offer significant benefits in terms of availability, cost effectiveness, and capacity.

In this talk, Sean O'Connor (Lead Application Developer, bitly) explains the challenges of building distributed systems and practical strategies to overcome these challenges. Specifically, Sean speaks to the benefits of Service Oriented Architecture, using asynchronous streams, scaling, dealing with failure, and monitoring.

This talk was presented at BaconConf.

Small jehiah Jehiah Czebotar on

Bitly - Realtime Distributed Message Processing at Scale with NSQ

This talk is by Jehiah Czebotar, Lead Engineer at Bitly, and Matt Reiferson, a Software Engineer at Bitly, recorded at the eBay NYC offices. This talk was recorded at the NYC Data Engineering meetup at Ebay NYC.

Matt and Jehiah will be talking about NSQ, their open sourced project  that solves an issue of  realtime distributed message processing, designed to operate at bitly’s scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.

Unknown author on

NSQ: Realtime Distributed Message Processing at Scale

(Contributor article by Bitly, originally appeared on the Bitly Engineering Blog)

NSQ is a realtime message processing system designed to operate at bitly’s scale, handling billions of messages per day.

It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.

Operationally, NSQ is easy to configure and deploy (all parameters are specified on the command line and compiled binaries have no runtime dependencies). For maximum flexibility, it is agnostic to data format (messages can be JSON, MsgPackProtocol Buffers, or anything else). Go and Python libraries are available out of the box.

This post aims to provide a detailed overview of NSQ, from the problems that inspired us to build a better solution to how it works inside and out. There’s a lot to cover so let’s start off with a little history…

Join Us