Defining the Right Business KPIs

Today’s businesses are numbers-obsessed, yet some of them are measuring the wrong things. Despite the universe of key performance indicators (KPIs) that can be measured, only a fraction matter to any specific organization. But which metrics are the right metrics?

“The right” metrics are those that align with and advance an organization’s goals. The wrong ones serve as barriers to progress. Telling the two apart isn’t always easy, though, because assumptions can get in the way.

1. More is not always better

Modern data analytics solutions enable organizations to measure all kinds of things for predictive and historical purposes. However, the art of the possible can backfire when it detracts from what really matters to the business.

“I don’t think a company that has thousands of metrics or hundreds of metrics and thousands of performs a whole lot better,” said Doug Laney, Gartner VP and distinguished analyst. “In fact, it may perform worse than those who take a dozen real meaningful metrics and don’t confuse people.”

2. Assumptions can be dangerous

Experienced professionals may think they know which metrics matter most, but if they’re wrong, the result may be grave consequences.

For example, Continental Airlines and Southwest Airlines both wanted to reduce fuel costs. According to Laney, Continental Airlines incentivized its pilots to reduce fuel consumption which they succeeded in doing by flying routes slower than usual and turning down the air conditioning on flights. What the leadership failed to consider were the unintended consequences of the metric which were lower customer satisfaction, poor on-time performance, and the added cost of hiring ground personnel to help passengers avoid missed connections. Worse, frequent flyers defected to competitors. Eventually, the business was sold at a discounted rate to United Airlines.

Meanwhile, Southwest Airlines relied on global economic indicators and proprietary algorithms to hedge fuel prices.

3. 20/20 hindsight is less valuable than foresight

When business outcomes differ from intended outcomes, sometimes organizational leaders will admit that they measured the wrong things. But then what? Did they start measuring different things and see improvement or did they fail to think through the problem adequately again?

According to Laney, the people that should be defining metrics are from the business in partnership with someone who understands the data, because the people who are defining metrics may not know what data exists, particularly outside the organization.

“[You need to] be well-aware of what data actually exists, how it’s compiled and what its limitations are. A few companies have a training/certification program one must go through before [being] allowed to do self-service analytics. I recommend the same sort of approach to anyone designing metrics,” said Laney. “Get trained on the data, get trained on analytics before you start creating KPIs and then work with someone who can think through the scenarios and unintended consequences of measuring that.”

4. Think beyond humans

When people think about KPIs, many are still thinking in traditional terms which is metrics as applied to people and processes. What they’ve overlooked is the growing influence of machines.

“Machines are increasingly the primary consumer of data and analytics,” said Laney. “While they may not reason with human aptitude yet, they can handle and respond to an order of magnitude more data than we can and they can take advantage of more measurements than we can.”

As businesses become more digital, their metrics modeling efforts should increasingly focus on machines.

5. Consider the beneficiary

If actual and intended outcomes differ in a negative way, it may be possible to identify the cause of the problem by reverse engineering some of the actions that lead to the mistake. During the process, it might become apparent why people, processes or incentives were driven down a certain path.

“Very often you’ll come up with a metric that maximizes their own economic benefit which may not benefit the organization because the metric or combination of metrics wasn’t designed well,” said Laney. “Designing meaningful metrics is as much an art as it is a science, so trial and error involves experimentation.”

6. Consider other data sources

More organizations are supplementing their own data with external data so they can do analyses that weren’t possible before, such as the effect of a snowstorm on hotel bookings, and therefore staffing requirements. The problem is, most organizations have not assigned someone to the job of procuring data sources.

Doug Laney, Gartner

“Most organizations have people procuring office supplies but nobody procuring data supplies,” said Laney. “The result is companies are primarily focused on their own data and not enough on what’s happening around them.”

7. Don’t just measure change, affect it

Metrics should be actionable. Otherwise, they may be interesting but will have little business value.

“Each metric should have a pedigree linking it to the company’s mission. It should be tied to incentives so they’re affecting the desired behavioral change,” said Laney. “Another thing is companies spend a lot of money on generating metrics and reports, but they don’t really gauge whether that effort is cost-effective.”

Metrics need to be evaluated, in other words, and that evaluation should be an ongoing, periodic process. Customers, processes, business environments and the availability of data sources change so the metrics also need to change.

“Unlike a fine wine, metrics don’t get better with age,” said Laney. “Organizational metrics tend to be somewhat static and they continue to be measured the same way year after year to achieve some sort of consistency. That’s not realistic because the business environment is changing.”

Instead, organizations should have a process that periodically reviews metrics, their impact and their cost to develop and produce. The process should also identify the unintended consequences of adopting a metric and contemplate new and emerging data sources.

8. Out-of-the-box is average

Vendors save companies a lot of time and expense by pre-defining metrics and making those metrics available in their platforms and solutions. However, everyone using that product or that version of the product has access to the same features.

“If you want outperform others that are using that software, perhaps you should think about improved metrics,” said Laney. “Most of the metrics that vendors provide tend to be hindsight-oriented, and we don’t find a lot of hindsight-oriented metrics moving the needle for a business. We have maybe 500 examples now, almost none of them have to do with building a pretty pie or bar chart or a fancy dashboard with hindsight-oriented metrics. They more often have to do with leading indicators that are more diagnostic, predictive or prescriptive and baked into an application, not just something that’s reported on.”

Bottom line

Analytics isn’t something that should be bolted onto a business strategy, it should be an integral part of it. As part of that, organizational leaders should think more critically about the KPIs they’re defining and the impact those KPIs may have on the business, good and bad.

With the best of intentions, organizations sometimes define KPIs that appear to be in the best interest of the business, but the metrics are short-sighted. While the organization may “succeed” as defined by the metric, it may also simultaneously affect other changes to the business that may be as dramatic as business failure. To avoid such consequences, it’s wise to pair the business leaders defining metrics with the people who understand the opportunities and limitations of the data, so KPIs are not only aligned with business goals, they can advance them.