Why AI Metrics Matter for Performance and ROI

As a growing number of organizations embrace AI, many are discovering that getting a handle on the technology's performance attributes can be challenging, to say the least, particularly when it comes to generative AI. 

We've found that most IT leaders are not currently on the same page about generative AI key performance indicators (KPIs,) says Jim Rowan, AI market activation leader at business advisory firm Deloitte. "Given that the technology is rapidly evolving, and its adoption is exponentially accelerating, the KPI landscape is likely not mature enough to get a clear picture of how leaders are measuring value -- and how they should be measuring value." 

Still, as organizations increasingly invest in GenAI to power their digital transformations, business leaders are eager to demonstrate the value of these investments, Rowan says in an email interview. "Without the right measurements in place, individual tech investments can be undervalued or underfunded, wasting valuable resources, missing opportunities for growth, and fueling misaligned expectations." 

A Strong Foundation 

Measurement is the cornerstone of both good science and engineering, and AI is no different, says Eric Heim, chief scientist, AI division, at Carnegie Mellon University's Software Engineering Institute in an email interview. He notes that measuring provides evidence to support or refute AI performance claims. "In other words, if done correctly, it gives you insight into how AI will perform when used in the real world." What makes measurement vitally important for AI, Heim says, is that most modern AI does not have strong principles to fall back on if success can't be measured. 

Related:Measure Success: Key Cybersecurity Resilience Metrics

GenAI and KPIs 

GenAI is a complicated topic in terms of KPIs because it can be leveraged to bring value in many ways, says Shelia Anderson, CIO at supplemental insurance company Aflac, in an online interview. "Generally, we use two sets of metrics -- technology-aligned and business-aligned," she states. "There are multiple KPIs that are important to us -- we consistently have key business goals aligned to KPIs that are jointly owned with the business and show clear delineation of value as aligned to business expectations," Anderson says. "GenAI is changing the way we work, so measuring that impact is of particular importance." 

Organizations using GenAI to measure productivity should closely monitor expense reductions, says Barbara H. Wixom, a principal research scientist at the MIT Center for Information Systems Research (CISR). "Productivity improvements can be a useful KPI in the sense that it might be a leading indicator of future expense reduction, but expense reduction is the most important KPI," she states in an email interview. 

Related:Uniting IT and Business Professionals Through a Shared Language of Success

Meanwhile, organizations using GenAI to create new products or enhance the value of existing products should closely monitor the bottom line, says Cynthia M. Beath, an academic research fellow with MIT CISR, via email. Are sales, for instance, increasing more than expenses? "Customer satisfaction, customer retention, or reduced service expenses might be useful KPIs if your organization understands how changes in those KPIs translate into future bottom-line impact." On the whole, however, product profit and loss is the most important KPI, she says. 

Other Factors 

To maximize productivity, GenAI should be seamlessly embedded within business processes and systems, says Manisha Khanna, AI strategist at analytics software firm SAS. "Performance should be reliable, transparent and ethical, accelerating productivity and improving customer experience, while adding measurable value to stakeholders," she advises via email. "A top KPI for GenAI focuses on reliability, including validating and ensuring response consistency to known prompts." 

Related:Secrets To Digital Success Are Obvious, But Difficult To Execute

Validating and ensuring consistency in foundation models' responses to known prompts is essential for determining the reliability of foundation models and for contributing to a better user experience, Khanna says. "This is especially critical when GenAI is used in regulated markets or in high-risk use cases that affect humans and society." She adds that it's important to understand the value realized from GenAI investments. "That means getting visibility into the components and associated costs that span across a model provider, cloud infrastructure and application provider." 

Gaining Visibility 

It’s important to understand the value realized from GenAI investments, says Alain Biem, chief data science officer at life insurance firm New York Life, in an email interview. The top KPI for GenAI is return on investment (ROI), he states. "ROI summarizes the targeted business value -- revenue, productivity, or cost saving -- versus the deployment cost -- development, maintenance, and opportunity cost." Biem notes. "In terms of a leading KPI, ROI is the best assessment of the overall value of technology, its adoption, and impact to the company bottom line." 

Final Thoughts 

Practically speaking, AI is costly, yet by itself it guarantees no payback, Wixom says. "Like any tool, AI must be used for it to pay back the investment." 

Heim, meanwhile, believes that AI metrics application should be guided by whatever is most important to its user. "For instance, for question answering, factualness is important," he notes. "For fantasy writing, factualness is not as important."