Continuous Delivery is Mainstream

Today, numerous “Web-scale” organizations (notably Google, Amazon and Facebook) practice continuous deployment, with real-time production monitoring and a lot of exploratory testingtaking place in production as well.

Here are a dozen articles about Web-scale continuous deployment in the enterprise

After reading these links it should be clear that Continuous Delivery is Yet Another Mainstream Software Methodology, just like Agile and Waterfall. Please reach out to me on Twitter if you think you can still make the case — in light of this evidence — that Continuous Delivery is still “untried” or especially risky ;-)

One other thing to take note of here: almost all these articles areold. Some were written 2 years ago or more. Rapid iterative development may or may not work for a specific organization — but the general problem of implementing Continuous Delivery is clearly well-solved and has been so for a while now.

Google practices Continuous Delivery

  1. At Google, 15,000 engineers work from the HEAD revision of a single Perforce trunk.

  2. 50% of the code will be changed in any given month.

  3. Google’s test infrastructure is legendary and they’ve written a comprehensive book about how they perform QAwhile continuously releasing.

  4. They’ve also put a lot of effort into scaling Perforce.

Here is a fantastic deep-dive into Google’s deployment pipeline:


Amazon practices Continuous Delivery

  1. At Amazon, new code is deployed to production at a staggering rate of once every 11.6 seconds during a normal business day.

  2. That’s 3,000 production deployments per day.
  3. "]
  4. They’ve invested an enormous amount of time and moneyinto creating an architecture that facilitates small, orthogonal, frequent code pushes.

Here’s Amazon’s Jon Jenkins breaking down their deploy stats at the O’Reilly Velocity conference:


Facebook practices Continuous Delivery

  1. At Facebook, each of 5,000 engineers commits to trunk HEAD at least once a day and the code at trunk HEAD is pushed to production once daily.

  2. Facebook has no dedicated QA team. All responsibility for testing rests with the software engineers.

  3. They’ve invested heavily in infrastructure that provides zero-downtime deployment at Facebook scale.

Here’s a snippet from Facebook’s release manager, Chuck Rossi, going into detail about how Facebook engineers balance their experiments against the risk of breaking some fundamentally important part of the site:

"Every work day Facebook is safely updated with hundreds of changes including bug fixes, new features, and product improvements. Given hundreds of engineers, thousands of changes every week and hundreds of millions of users we have worldwide, this task seems like it should be impossible. In this tech talk, Chuck Rossi will dig into the tools and processes built by our Release Engineering team that make it possible to push daily updates to the site."

UPDATED January 21, 2014

I meant the adjective “mainstream” in the sense of “not dangerous.” For example: “The Ramones are so mainstream, I only listen to Norwegian Death Metal.” Based on the large amount of feedback I have received, this is not everyone’s default definition of “mainstream.” Hopefully adding this paragraph to the post will clear up such ambiguity for future readers.

Once again, I did not mean to imply that everyone is doing CD. Everyone is not doing CD! But, CD is no longer the risky experiment it was in 2010 when Chad Dickerson hired me to help scale the CI system at Etsy (which is how I got involved in this whole discussion in the first place). Today CD is a mature option and I think it is the best option available. But there are certainly other ways to build software and lots of people use those other methodologies to great success.

I apologize for once again having slightly confused a small portion of the Internet and ultimately starting a couple of interesting if off-topic conversations. Now here is a kitten GIF to atone for my actions and let us never speak of this again.

Wow that kind of turned into a rant. Sorry. Anway — as I was saying: Google, Amazon and Facebook all are using very aggressive Continuous Delivery workflows and have been doing so for years.

Weekly, daily or even hourly production deployments are a commonly-practiced, enterprise-scale software development life cycle. (Editor’s note: in retrospect we can see how “commonly” might be taken to mean “universally” in this sentence. That was not our intent. In our opinion, no methodology is universally practiced. Many successful and competitive organizations still practice Waterfall. See also diseconomies of scale.)

Original article here