Measure DevOps for Efficiency, not Just Speed

Organizations with successful DevOps deployments “get their developers to work faster and their operations teams to work more efficiently, and bring it all together,’’ according to Jay Lyman, principal analyst at 451 Research.

When DevOps first hit the enterprise space a few years ago, “it was all about speed, and developers were moving faster," Lyman said. That included faster iterations and less time "mucking around with system management and the infrastructure’’ for both app testing and production deployment, Lyman said. While speed remains a big driver, “efficiency has become more of a priority.”

One way to improve DevOps is to focus on more efficient cloud computing, he explained. Because DevOps is so often delivered via the cloud, better cloud utilization and less overspending translates into broader and more successful DevOps deployments.

Also, more organizations in general are using metrics to measure individual IT-related processes, programs, functional areas, and individuals. Updating these metrics to measure and improve the overall workflow helps a matrixed organization transform to a DevOps culture, while still having a matrixed model, according to IT Revolution's report, Making Matrixed Organizations Successful with DevOps, which resulted from the DevOps Enterprise Forum held in 2017.

The report suggests companies “transform” certain metrics, get rid of them in certain cases, or update them. From the report: “Current metrics, such as number of lines of code produced by a development team, directly hinder the overall goal of improving the value stream. Either transform this metric to include lines of code created per feature and for automated testing (for example), update the metric to include the required focus on the end-to-end value stream, or get rid of a lines-of-code metric altogether and bid based on feature size for design, code, test and deployment."

Another suggestion is to add metrics for measuring work in progress (WIP) and rework, which are functionally agnostic. Another metric, defects found in QA/Test, needs to be changed to push the finding of defects earlier in the cycle and to reduce the defects found so late in the cycle, the report said.

{image 2}

One of the top DevOps trends now and looking ahead is a focus on management and measurement, because speed alone does not guarantee success, according to the Forrester Research report, Six Trends That Will Shape DevOps Adoption in 2017 And Beyond. When asked how often their team(s) release applications, respondents to Forrester’s Q1 2017 Global DevOps Benchmark Online Survey said current average speeds vary, including daily releases (8%) one release per week (25%), one per month (32%), and one per quarter (22%).

“Top speed alone does not indicate a mature DevOps organization,” the trends report stated. “Instead, think through what speed is necessary for the existing context and purpose of a specific service or application.”

Why uptime matters for DevOps

Uptime is another key metric, generally used for operations, which directly impacts the ability to improve the value stream, the report noted. (Forrester defines the "value stream" as the activities organizations engage in to go from a business idea to delivering a solution that meets a business need).

Although focusing on uptime in general helps support the overall goal of improvement, this metric also should be updated. It needs to include uptime of specific business functions “to allow the functional areas to manage their teams and measure appropriately, but also improve the throughput of the value stream,” the report said.

Each functional area should also review any metrics that are related to the overall value proposition and not just its specific piece of the pie. “Functional area leaders need to agree on these new metrics to ensure they support the overall business goals while supporting the improved flow for delivering business value,’’ the report advised.

Cross-functional teams should take the lead here, to ensure the measurements for each area provide value as well as drive the right behaviors for successful DevOps transformation.

Metrics in the real world

From the time it was founded in 1970, Glidewell Laboratories, a provider of dental products and services, has been fixated on innovating. That approach sat well with Robert Edwards, who came on board as software manager more than three years ago and wanted to get his group focused on microservices, agile, and constant software deployment.

Robert Edwards, Glidewell

Docker “was our first introduction to real, true DevOps,” said Edwards, now director of software engineering. The old waterfall approach was limiting, he said. “If you were starting a new app maybe it would get deployed in three to four months -- if not longer -- and we wanted to be more rapid because leadership on the business side wanted to quickly test out theories.”

Another key goal for his group was to become more responsive to the company's digital customers.

Edwards admitted his DevOps team hasn’t been heavily focused on metrics. “What we do know is we deploy to production at a more constant rate,’’ he said. “Now we’re doing maybe 10 to 15 deployments per day of new components,” and his app development team is also able to prototype more quickly – sometimes within 24 hours, he said.

IT is also continuously deploying its application platform Elektra, which was revamped and integrated onto a new platform -- Amazon Web Services.

“If we took the waterfall approach, it wouldn’t be ready to use for some time,” as much as a year or more, Edwards pointed out. “We could use parts of it right away and hook in different apps more rapidly” using microservices. The IT group and the one-person DevOps team are constantly updating Elektra and the apps that are part of it.

Edwards added that one of the limitations his firm has with DevOps right now is there's only one person dedicated to the practice, but the company believes in it and Glidewell is planning to hire at least one other DevOps person in 2018. “We realize there’s a lot more to DevOps and we want to improve,” he said.

In the end, formal metrics may not matter when you can constantly update and iterate. “The ability to develop and deliver software efficiently and accurately is a key differentiator and value driver for all organizations … if you want to deliver value, no matter how you measure it, DevOps is the way to go,’’ according to Puppet's 2017 State of DevOps Report.

DevOps barriers

For all the benefits of DevOps, there are still issues hindering the ability for IT departments to maximize the value of this approach. Security, compliance/regulatory issues, lack of skills, limited budgets, and application complexity are still challenges, industry experts say.

Like at many companies, change management remains something of a roadblock at Glidewell, and it doesn’t always fit with the concept of deploying 10 new components a day, said Edwards. Traditional signoff and the “gated approach,” where code would go from development to QA to production, is hard to maintain with rapid app development, he said.

[Read more about trends in enterprise DevOps adoptionfor 2018 in The World of DevOps Tools Can Make a Newbie's Head Spin and Implementing DevOps: Mind the Details.]

There is also fear when roles change, he said. Telling one of his systems engineers to deploy automation “feels a little uncomfortable,” he explained. “Or the expectation is to have something in production quickly, and that’s uncomfortable, because they’re used to the long view around things.”

A lot of education is needed, he said, so everyone can see how they can contribute to software deployment. “Tech people aren’t that different from end users,’’ Edwards said. “When you’re changing processes, people get concerned.”

The process becomes “really ripe for failure at certain points if the instructions aren’t well produced or understood. That’s where we struggled,” he added. But once those challenges are overcome, repeatable deployment and testing have made software development and deployment go much more smoothly, he said.

On tap for this year, Edwards noted, is to bring in tools to support continuous integration and deployment, and moving from “ideation to code check-in to final production deployment; getting that visibility all the way through that pipeline.” 

“DevOps implementations are no longer one-offs and in the shadowy IT area,’’ says Lyman. “DevOps is now part and parcel of enterprise IT.”

Just a Bit of Advice

Metrics matter in DevOps, but so do other factors

Beyond metrics, another key to a successful DevOps program is to look for opportunities to either merge teams or move work to fewer teams when possible. Some companies have had success merging development and quality assurance teams, so dev and test cycles can be completed more quickly and efficiently.

Additionally, IT Revolution's report -- Making Matrixed Organizations Successful with DevOps -- recommended that enterprises “be sure to avoid creating teams that are too large, which creates handoffs and context breaks within a team.”

Transformational leadership is another essential ingredient for implementing technologies and processes that bump up developer productivity, reducing code deployment lead times and supporting more reliable infrastructures, according to Puppet's 2017 State of DevOps Report.

Automation is also important, yet teams are not always good at estimating how automated they are, and some work is still manual when it comes to deployment and change approval processes, the Puppet report said.

Esther Shein is an independent technology journalist based in Massachusetts. She formerly was the editor in chief of Datamation.com and a senior writer at eWeek (formerly PC Week) magazine, and was a reporter for general news media outlets.