Avoid the Danger Zone of Metrics - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management // Big Data Analytics
02:00 PM
Dawn Parzych, Director, Catchpoint
Dawn Parzych, Director, Catchpoint

Avoid the Danger Zone of Metrics

Organizations love metrics, particularly those that show success, but it's crucial to define metrics that truly track progress toward corporate goals.

IT teams track thousands of metrics to make informed decisions for their organizations. Metrics are used to share information, understand the current state, judge successes or failures, identify anomalies, or predict the future. This all sounds great, but there is a dangerous side to metrics.

When metrics aren’t focused on the right thing, aligned across teams, or are gamed, their value decreases. Here are some well-worn patterns to help you determine if you’re entering a danger zone with your metrics:

Proxy metrics

What you want to measure and should measure -- compared to what you can measure -- isn’t always the same thing. Teams often fall back to using proxy metrics since there is no apparent way to measure what they truly want to track.

Vanity metrics

Proxy metrics can lead to vanity metrics. Vanity metrics feed into our desire to demonstrate value. They are often large numbers that always seem to increase, and sound impressive. One of the most famous vanity metrics is McDonald’s quoting “billions served.” Vanity metrics tend to be cumulative and don’t help us understand usage patterns. When you pay attention to vanity metrics, you lose sight of what really matters. For example some companies track daily active users (DAUs) to measure growth. DAUs don’t reveal much about what the user is doing, how much time they are spending on a site, if they are satisfied, or how many users are lost. If the goal of having more users is to have them download or purchase a subscription, DAUs won’t tell you whether this goal is being met. More users may not necessarily equate to more downloads or subscriptions purchased.

Summary data

Means and medians are easy to understand but they often don’t tell the whole story. They hide the presence of anomalies and outliers. Often we can learn more from the anomalies and outliers than we can from the average events. A common metric used to track performance of incident resolution is mean time to resolve (MTTR). This doesn’t show whether all incidents are resolved within the same timeframe, or if some incidents take a few minutes while others take much longer. To improve this metric there needs to be an understanding of the distribution of measurements. Two great visualizations of this are available from AutoDesk and FlowingData

Selecting the metrics that truly reflect what matters to your business is critical to IT and business success. Some tips to help you choose the right metric include:

Align your metric with larger organizational goal. If you want other people to take notice of your metric, portray it in a way that matters to them. The three things that matter to an organization are revenue, risk, and costs. How does your metric align with one of these three objectives? Ask yourself “so-what?” to elevate why a metric matters. Individual and team goals may focus on culture, collaboration and sharing. These can be used to measure efficiency, effectiveness, quality and velocity. These in turn bubble up to measure customer and business value.

The Ideal IT Metrics Funnel

So, align these metrics to the bottom-line concerns of the organization. Ask: Why should others care? Examples: If mean time to repair (MTTR) decreases, customers are more satisfied and will spend more money. If MTTR increases, customer and employee satisfaction decreases. Employee burnout leads to higher recruitment and training costs.

Track actionable metrics. What action can be taken to make a metric move? A metric should show you what is going right or wrong and how to improve. If nobody knows what can be done to change the trajectory of a metric, it is not useful.

Look at the big picture. While simplicity in metrics is important, don’t focus on a single metric. Even if you’re not using a vanity metric, hyper-focusing on a single metric or viewing it out of context can still lead to negative consequences. For example, reporting on the number of software deployments can show growth and productivity, but if number of incidents increases at the same time are things really improving?   

Lead to growth. Healthy competition is good, but metrics should not be used to pit teams against one another or shame individuals. If you are trying to create a culture of sharing, mentoring, or constructive peer reviews, using metrics that deem people as winners and losers are counterintuitive. We need to be able to compare metrics across times, across teams, or across systems to identify what’s working and what needs to change. Look for the similarities and differences without assigning blame.

We operate in a world of ever-increasing data, so it’s important to remember that metrics aren’t set and forget. They should constantly be evolving and changing as your team and organization changes. The metrics that matter today may not matter 6 or 12 months from now. Metrics should guide us on a path towards continuous improvement. That path will likely be a windy road with ups, downs, and potential detours. That’s OK. When a metric becomes a target we see through rose-colored glasses it ceases to be a good measure. Metrics can open lines of communication and lead to alignment across teams when they are actionable, comprehensive, and comparative. Finding the metrics that exhibit these qualities may not be easy, but it is worth it.

Dawn Parzych
Dawn Parzych

Dawn Parzych is a director at Catchpoint, a digital experience intelligence company. She has  deep expertise in topics relating to the psychology of IT and frequently researches, writes, and speaks about trends related to application performance, user perception, and how they impact the digital experience. In 15+ year career, Dawn has held a wide variety of roles in the application performance space at Instart Logic, F5 Networks, and Gomez.


The InformationWeek community brings together IT practitioners and industry experts with IT advice, education, and opinions. We strive to highlight technology executives and subject matter experts and use their knowledge and experiences to help our audience of IT ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Oldest First  |  Newest First  |  Threaded View
User Rank: Apprentice
8/30/2017 | 4:40:40 AM
Making Sense
Great to see a more holistic and sense making approach to metrics. You hit the correct spots for organizational alignment and strategy. 

In the transition phase of replacing legacy metrics to power metrics, I suggest introducing reporting by exception and value reviews of existing metrics becomes a 12 monthly operation for the business.


User Rank: Apprentice
9/7/2017 | 8:16:30 AM
Only half the story
It's important to realize that two different types of metrics are needed. The article deals with result metrics, which the article correctly points out too often are not appropriate. In addition, though, we need metrics that help us understand, maintain, and address causes of those correctly defined and measured results.
User Rank: Apprentice
12/19/2018 | 1:07:48 AM
Pending Review
This comment is waiting for review by our moderators.
Can Cloud Revolutionize Business and Software Architecture?
Joao-Pierre S. Ruth, Senior Writer,  1/15/2021
10 IT Trends to Watch for in 2021
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/22/2020
How CDOs Can Build Insight-Driven Organizations
Jessica Davis, Senior Editor, Enterprise Apps,  1/15/2021
White Papers
Register for InformationWeek Newsletters
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you.
Flash Poll