DevOps Demands You Fix Twice - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IT Leadership // CIO Insights & Innovation
09:06 AM
Rajat Bhargava
Rajat Bhargava

DevOps Demands You Fix Twice

If you've taken time to get everything just perfect, you haven't truly adopted the rapid iteration, lean product development ethos.

My friend and colleague Richard Miller, CTO of genealogical search engine Mocavo, likes to say, "Fix everything twice." When I first heard him say that, I didn't quite understand what he was getting at. Why not just fix a problem properly the first time so it doesn't come up again? He replied that it's just like carpentry -- measure twice, cut once. Richard was starting to mix metaphors, and he lost me.

What Richard's talking about are DevOps and lean startup principles. Organizations today -- especially startups -- are moving too fast to get everything right all the time. Add to that how quickly technology is shifting, and Richard's "fix twice" adage makes perfect sense, even for enterprises.

DevOps is an extension of Agile that involves the rest of the organization in rapid iterative releases, focused on driving increased customer value. Quicker turns on the product mean bringing new capabilities -- presumably features customers are asking for -- to market faster, fostering a competitive business. DevOps uses a more integrated team environment and automation to deliver on accelerated product releases

One result of iterating often is that organizations don't feel the need to build far beyond their scale, because they know that they can come back in the next sprint and up the ante. Compare that to the old model, where you have only a couple of shots to get things right. You can't really go back and attack shortcomings after the fact. Shorter release cycles mean, in general, that you can adjust quickly.

All that said, when product releases really get rolling, some things are going to break. Fortunately, DevOps is about things breaking. If you've taken time to get everything just perfect, you haven't truly adopted the rapid iteration (or lean product development) ethos.

DevOps means speed, our 2014 survey says.
DevOps means speed, our 2014 survey says.

If your organization can build and release 12 times in a year, will you have a better, more useful product than if you had released only at the end of the year? In almost all cases, yes. The organization that can release more quickly will win. It will be able to adjust based on feedback so that it's building the product that most closely matches what customers are looking for. Additionally, it's going to hear about problems from customers continually versus waiting to find out how the product worked, or didn't.

This is where "fix twice" comes in.

In a DevOps culture, IT teams are moving fast. Solutions to problems need to happen just as quickly. An application or site goes down? Put a Band-Aid on the problem and stitch it up later. That's fix number one -- quick and dirty. The goal is to stabilize the issue with the least amount of effort. The team is generally working under pressure and duress. It's difficult for those involved to think about anything other than getting things operational again.

[If you realize that mobile security means more than ensuring users don't download malware-bearing games from the Android store, take our 2014 survey and enter to win a 32 GB Kindle Fire HDX.]

The real magic happens in fix number two. This is where the team goes back, studies the problem, and takes action to fix it permanently -- or as close to permanently as is practical at the time.

A DevOps team under pressure to rectify a major issue doesn't exist in an environment conducive to strategic evaluation and problem solving. By putting a patch on the initial problem, you bought breathing room. Fix two gives the team an opportunity to find the best long-term solution.

People can debate different approaches, spike on the problem, even try a few different tactics before settling in on the right long-term strategy. Ironically, a team that has experienced failure can better evaluate the problem and focus on creating an ideal, longer-term solution than one that is trying to solve a problem in advance.

Perhaps the most significant challenge with this approach is that most organizations don't have the discipline to get to fix number two. Fix number one took the pressure off. As long as the Band-Aid stays stuck, there's little incentive to circle back. With DevOps and IT folks stretched so thin, it's often difficult to pick up our heads and think about longer-term problems.

Unfortunately, with technology, by the second or third time a problem rears its head, the situation is generally pretty dire. It will take a lot more than Band-Aids to keep things together.

The best DevOps organizations don't let the same problems recur. They schedule time during the sprint cycle to permanently address core issues, and as a result have longer-term fixes, which translate into greater stability.

So while Richard likes to quip "fix twice" often, the real message is not just about solving specific problems. It's about building a methodology and inculcating the discipline to execute on that method. DevOps is at the core of this approach, as is the ability to move fast. For IT organizations that want to be the best, fix failures twice.

Female IT leaders attending the InformationWeek Conference can join Editor-In-Chief Laurianne McLaughlin and Rebecca Kaul, President of the UPMC Technology Development Center, for a peer networking breakfast. Then join your peers at our Interop Women in Technology Panel & Luncheon for an open forum to discuss how to advance in an IT organization, keep your skills sharp, build fruitful relationships with colleagues, learn effective dispute resolution techniques, and build a mentoring network. Space is limited to 50 participants.

Rajat Bhargava is co-founder and CEO of JumpCloud Inc., a provider of server management and security tools for DevOps and IT professionals. An MIT graduate with two decades of experience in industries including cloud, security, networking, and IT, Rajat is an eight-time ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Lorna Garey
Lorna Garey,
User Rank: Author
3/18/2014 | 6:37:01 PM
Re: OHIO principle
Kevin, I guess I'll buy that. I do think it'll be a telling indicator of the maturity of an org's devops practice as to how religiously they do circle back. One a fire is out, it's human nature to move on.
Lorna Garey
Lorna Garey,
User Rank: Author
3/13/2014 | 11:20:56 AM
OHIO principle
This flies in the face of the venerable "OHIO" (only handle it once) principle that says, "once you've taken time to understand a problem you should just fix it." Otherwise, you must circle back and redo that work of garnering insight. And of course, time is no friend of memory.

Would you recommend spending time during the first fix to thoroughly document the problem?
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

11 Things IT Professionals Wish They Knew Earlier in Their Careers
Lisa Morgan, Freelance Writer,  4/6/2021
Time to Shift Your Job Search Out of Neutral
Jessica Davis, Senior Editor, Enterprise Apps,  3/31/2021
Does Identity Hinder Hybrid-Cloud and Multi-Cloud Adoption?
Joao-Pierre S. Ruth, Senior Writer,  4/1/2021
White Papers
Register for InformationWeek Newsletters
Current Issue
Successful Strategies for Digital Transformation
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Flash Poll