Continuous Delivery: 7 Hard-Earned Lessons for Getting it Right - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

04:00 PM
Russell Smith, CTO of Rainforest
Russell Smith, CTO of Rainforest

Continuous Delivery: 7 Hard-Earned Lessons for Getting it Right

As organizations move toward continuous integration and delivery of applications there are multiple ways in which things can go wrong.

It seems laughable now, but do you remember when Internet Explorer was updated every two years? Times have changed. Development cycles are approaching warp speed as companies deal with the competitive pressure to update their applications and websites as frequently as possible.

Facebook has said it releases new code into production twice a day -- and that was three years ago. Google Chrome is fast approaching Version 60. Most businesses don’t need to release new features so frequently, but everyone wants to be quicker. A Forrester study found that 51 percent of business leaders want new software capabilities to go from concept to delivery in fewer than six months, but that most developer teams can’t accommodate that pace.

Organizations that need to move fast have adopted continuous integration and delivery as their model. With CI/CD, new features are released as soon as they’re ready instead of following some arbitrary release cycle. We’ve done continuous delivery at Rainforest for a few years now and I can attest to the benefits, but it’s tricky to get right. The CI/CD approach requires new tooling and a new mindset on the part of your team, and if the process isn’t managed well you can easily end up releasing broken software.

Here are 7 hard-learned lessons from our engineering team for making CI/CD a success.

Release in small increments. At Rainforest, we often release changes a few lines at a time. Rather than roll bug fixes and small features into a larger update, we push each individually into production as soon as it’s ready. These are still tested before release, but having the smaller chunks means that if something doesn’t work, it’s much easier to see what broke, and rolling back is trivial, fast and low risk.

Use pull requests. Pull requests allow you to separate the features you’re working on, update them, and then merge them back into the releasable branch. Github is your friend here. Our code always starts out on a feature branch -- committing directly to the master branch leaves too much room for error. When the author is convinced the code is ready to ship, they make a pull request. When unit tests have been run and the reviewer is satisfied, the code is merged to the develop branch. An automatic release from develop through to production happens from here out, covering QA, unit tests, migrations and releasing.

Find the right reviewer. Code reviews, while awesome, can turn into a major bottleneck if they're “hierarchical,” meaning each review has to come from a more senior developer. At Rainforest, our code review is almost entirely peer-based to keep things moving fast. If code requires additional scrutiny -- if it’s security-related, for instance -- we flag it for inspection automatically by at least two developers, including the CTO.

Benchmark for performance. While most checks involve quality and bug prevention, if performance is critical for your app, add an automated benchmark suite to your CI build. This is a great way to avoid inadvertently introducing performance regressions, as well as to make sure you hit your overall production performance targets.

Continuous QA. If you’re doing continuous delivery, you need continuous testing. Otherwise you’re constantly at risk of breaking things, or waiting for your QA team to catch up with your developers. Whatever you choose, be it automation or super-fast crowd-sourced testing, make sure it fits with whatever you think is reasonable for your release process. Most companies aim for 30 minutes end-to-end.

Automate wherever possible. This should go without saying, but automate as much of the process as you can to keep things moving quickly. This stops things getting dropped and improves consistency. We’ve automated processes for code review requests, unit testing, benchmarking, and, where possible, QA. Manual testing, unless it’s extremely cursory, can’t keep up with CI/CD. You must automate, or find some other method of comparable speed.

Build the right stack. Pick tools that integrate well with each other and fit into your team’s workflow. We use Github for hosting our code, CircleCI for CI/CD, and Heroku for our infrastructure. We also use Circlemator, an open source tool we released ourselves, and Fourchette, which has now been implemented in Heroku.

Russell Smith, Rainforest
Russell Smith, Rainforest

Clearly, CI/CD isn’t a magic bullet, but done right it can improve the quality of your software. Releasing code in small chunks is the least risky way to deploy, since fewer changes means there’s less chance of unforeseen bugs -- and easier debugging when bugs do appear. But it’s a big shift for dev teams, even compared to agile. I hope these tips will help to make it a success at your organization.

Russell Smith is the CTO & co-founder of Rainforest QA, the industry's only AI-powered crowdtesting platform that helps agile and continuous delivery engineering teams move faster. His specialties include development, developer workflow, DevOps, Linux, Debian, CI, benchmarking, profiling, bug fixing, performance, scalability, ops planning, capacity planning and modeling.

The InformationWeek community brings together IT practitioners and industry experts with IT advice, education, and opinions. We strive to highlight technology executives and subject matter experts and use their knowledge and experiences to help our audience of IT ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Becoming a Self-Taught Cybersecurity Pro
Jessica Davis, Senior Editor, Enterprise Apps,  6/9/2021
Ancestry's DevOps Strategy to Control Its CI/CD Pipeline
Joao-Pierre S. Ruth, Senior Writer,  6/4/2021
IT Leadership: 10 Ways to Unleash Enterprise Innovation
Lisa Morgan, Freelance Writer,  6/8/2021
White Papers
Register for InformationWeek Newsletters
2021 State of ITOps and SecOps Report
2021 State of ITOps and SecOps Report
This new report from InformationWeek explores what we've learned over the past year, critical trends around ITOps and SecOps, and where leaders are focusing their time and efforts to support a growing digital economy. Download it today!
Current Issue
Planning Your Digital Transformation Roadmap
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Flash Poll