From Measurement to Insight: Put DevOps Metrics to Work - InformationWeek
IoT
IoT
DevOps
Commentary
3/14/2018
07:30 AM
Anders Wallgren
Anders Wallgren
Commentary
Connect Directly
Twitter
RSS
50%
50%

From Measurement to Insight: Put DevOps Metrics to Work

When you can measure the results of your DevOps program, that's where the magic happens.

Working in the software delivery industry today is not for the faint of heart. Blink, and your competition has already passed you by with a product delivered that much faster and better. However, it is possible to stay on pace with even the best in the business by adopting the best practices and principles associated with DevOps.

While there are many keys to a successful transformation journey, arguably one of the most critical is creating feedback loops through measurement. Unfortunately, measurement and data collection methods easily become time consuming – in a world where DevOps is pushing speed – and, as a result, cycle and response times may suffer. The good news is that there are ways to turn metrics into action that will optimize the deployment pipeline to produce real-time results.

Image: Shutterstock
Image: Shutterstock

I recently sat down with Dr. Nicole Forsgren, CEO and Chief Scientist at DORA and co-author of the book “Accelerate.” Together we discussed the importance of DevOps and Agile, metrics that matter to the business, and how data-driven decisions can guide continuous improvement efforts. Here is an overview of our key conversation points:

Why measure?

Let’s break down the importance of metrics. From my perspective, DevOps is really the art and science of continuous improvement. Organizations are all trying to deliver better software but dealing with high costs in terms of risk and time to release. Today, many of these processes are operated manually, using non-integrated tool chains or islands of automation that are separated by wide slots of manual hand-offs.

This is where metrics come in. In order to understand where you want to go, you need to understand where you are at right now, and the only way to do that is to measure. It is critical to start collecting baseline metrics off the bat – whether its release time or mean time to recovery (MTTR) – to statistically visualize where improvements are being made and where bottlenecks may still exist. The metrics that matter most might vary from company to company, but the right ones for your company will focus on what brings true business value. Nicole also shared her thoughts on this topic:

Nicole: I've been lead investigator on the State of DevOps report for the last four years. This report is presented by DORA (myself, Jez Humble, and Gene Kim) and Puppet. What I've noticed is that the industry continues to change, it continues to evolve. We've been getting better year, over year, over year, and what's important is that we continuously improve. We capture metrics in an effort to strive to get better every year and not just set our sights on a destination. Because if we just try to point to a destination, once we arrive at that destination and we stop, the rest of the industry will continue to get better and they will pass us by.

It's super important to embrace the right metrics and move forward because even a bad baseline can be powerful, because it can be used to communicate the importance of resource investment. Even if we suck at something, that's fine. Embrace the bad…

Always keep metrics in mind as you're fixing problems in order to learn how to improve and grow, even if those improvements are small or don’t work at first.

Challenges with metrics

The rapid rise of automation has certainly been a double-edged sword. While it’s helping organizations keep up with digital transformation and quickening software delivery paces, it also means that bugs and other errors are harder to catch and can be sent to production more easily. For example, you might fix a bug in one line of code, but that particular bug might appear in 14 other lines of code, add in automation and the end product can leave customers unsatisfied. Even something as deceptively simple on the surface as MTTR [Mean Time to Repair] actually turns out to be a fairly robust metric to measure when automation is thrown into the mix. This is where establishing metrics and goals upfront can play a critical role in catching potentially dangerous errors from being deployed into production. Metrics help teams along the software delivery pipeline understand what is important to the business, aligns them to the same goals and understand where bottlenecks exist. Nicole added her thoughts, citing challenges with automation.

Nicole: Even with automation, it's difficult. There may be errors or something else going wrong in your code. When you can get visibility into something, that’s valuable and more automation can help with that.  

Metrics that matter

We say “everything as code” often at Electric Cloud. Infrastructure as code. Pipeline as code. Testing as code. You don't have snowflake environments. Everything is change managed. Self-service automation tends to be a big one. That's a really big one for idle time. In our experience, a big chunk of idle time comes from waiting for environments.

I think, the idea that we'll have lots of people doing manual testing, that time has come and passed. I think we all need to be more test writers and test runners these days because the test run will get automated pretty soon... it's automate, automate, automate when it comes to all the things that I like to talk about. Nicole also shared her perspective on metrics:

Nicole: Implementing system-based metrics and survey-based metrics together can be a nice complement. They complement each other. So what you can do is use people to give you insight into how your systems are running.

People can tell us if we're not using version control. People can tell us how often we deploy. People can tell us approximately what our MTTR is, right? People can give us reasonable insights to the particular health status of our deployment pipeline. So we can use people-based, survey-based measures. It can range from non-scientific to advanced scientific measures. DORA does have scientific survey-based measures, but then we can use those to help fill out and complement our system-based measures.

The heart of DevOps and Agile is fast feedback. Measuring your pipeline and collecting information about performance and quality is great, but is time consuming. This means longer cycle times and slower responses are inevitable. However, the magic happens when you can turn that telemetry into actionable insights in order to optimize end-to-end flow. Just remember: When embarking upon the journey to better measurement, you first have to have an idea of what questions you want to answer before you dive head first into the data sets.

[Anders Wallgren will return as a presenter at Interop ITX 2018 with his session DevOps and ChatOps in the Enterprise on May 3.]

 

Anders Wallgren brings with him more than 15 years of in-depth experience designing and building commercial software. Prior to joining Electric Cloud, he held executive positions at Aceva, Archistra, and Impresse. He also held management positions at Macromedia, Common Ground ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
2018 State of the Cloud
2018 State of the Cloud
Cloud adoption is growing, but how are organizations taking advantage of it? Interop ITX and InformationWeek surveyed technology decision-makers to find out, read this report to discover what they had to say!
News
IT Budgets: Traditional Still Bigger than Cloud
Jessica Davis, Senior Editor, Enterprise Apps,  9/20/2018
Commentary
Building a Smart City Doesn't Have a Common Blueprint
Guest Commentary, Guest Commentary,  9/18/2018
Commentary
AWS vs. Azure: Users Share Their Experiences
Guest Commentary, Guest Commentary,  9/7/2018
Register for InformationWeek Newsletters
Video
Current Issue
The Next Generation of IT Support
The workforce is changing as businesses become global and technology erodes geographical and physical barriers.IT organizations are critical to enabling this transition and can utilize next-generation tools and strategies to provide world-class support regardless of location, platform or device
White Papers
Slideshows
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Sponsored Video
Flash Poll