Continuous Delivery: 7 Hard-Earned Lessons for Getting it Right

It seems laughable now, but do you remember when Internet Explorer was updated every two years? Times have changed. Development cycles are approaching warp speed as companies deal with the competitive pressure to update their applications and websites as frequently as possible.

Facebook has said it releases new code into production twice a day -- and that was three years ago. Google Chrome is fast approaching Version 60. Most businesses don’t need to release new features so frequently, but everyone wants to be quicker. A Forrester study found that 51 percent of business leaders want new software capabilities to go from concept to delivery in fewer than six months, but that most developer teams can’t accommodate that pace.

Organizations that need to move fast have adopted continuous integration and delivery as their model. With CI/CD, new features are released as soon as they’re ready instead of following some arbitrary release cycle. We’ve done continuous delivery at Rainforest for a few years now and I can attest to the benefits, but it’s tricky to get right. The CI/CD approach requires new tooling and a new mindset on the part of your team, and if the process isn’t managed well you can easily end up releasing broken software.

Here are 7 hard-learned lessons from our engineering team for making CI/CD a success.

Release in small increments. At Rainforest, we often release changes a few lines at a time. Rather than roll bug fixes and small features into a larger update, we push each individually into production as soon as it’s ready. These are still tested before release, but having the smaller chunks means that if something doesn’t work, it’s much easier to see what broke, and rolling back is trivial, fast and low risk.

Use pull requests. Pull requests allow you to separate the features you’re working on, update them, and then merge them back into the releasable branch. Github is your friend here. Our code always starts out on a feature branch -- committing directly to the master branch leaves too much room for error. When the author is convinced the code is ready to ship, they make a pull request. When unit tests have been run and the reviewer is satisfied, the code is merged to the develop branch. An automatic release from develop through to production happens from here out, covering QA, unit tests, migrations and releasing.

Find the right reviewer. Code reviews, while awesome, can turn into a major bottleneck if they're “hierarchical,” meaning each review has to come from a more senior developer. At Rainforest, our code review is almost entirely peer-based to keep things moving fast. If code requires additional scrutiny -- if it’s security-related, for instance -- we flag it for inspection automatically by at least two developers, including the CTO.

Benchmark for performance. While most checks involve quality and bug prevention, if performance is critical for your app, add an automated benchmark suite to your CI build. This is a great way to avoid inadvertently introducing performance regressions, as well as to make sure you hit your overall production performance targets.

Continuous QA. If you’re doing continuous delivery, you need continuous testing. Otherwise you’re constantly at risk of breaking things, or waiting for your QA team to catch up with your developers. Whatever you choose, be it automation or super-fast crowd-sourced testing, make sure it fits with whatever you think is reasonable for your release process. Most companies aim for 30 minutes end-to-end.

Automate wherever possible. This should go without saying, but automate as much of the process as you can to keep things moving quickly. This stops things getting dropped and improves consistency. We’ve automated processes for code review requests, unit testing, benchmarking, and, where possible, QA. Manual testing, unless it’s extremely cursory, can’t keep up with CI/CD. You must automate, or find some other method of comparable speed.

Build the right stack. Pick tools that integrate well with each other and fit into your team’s workflow. We use Github for hosting our code, CircleCI for CI/CD, and Heroku for our infrastructure. We also use Circlemator, an open source tool we released ourselves, and Fourchette, which has now been implemented in Heroku.

{image 1}

Clearly, CI/CD isn’t a magic bullet, but done right it can improve the quality of your software. Releasing code in small chunks is the least risky way to deploy, since fewer changes means there’s less chance of unforeseen bugs -- and easier debugging when bugs do appear. But it’s a big shift for dev teams, even compared to agile. I hope these tips will help to make it a success at your organization.

Russell Smith is the CTO & co-founder of Rainforest QA, the industry's only AI-powered crowdtesting platform that helps agile and continuous delivery engineering teams move faster. His specialties include development, developer workflow, DevOps, Linux, Debian, CI, benchmarking, profiling, bug fixing, performance, scalability, ops planning, capacity planning and modeling.