Businesses Apply New Metrics In Measuring IT's Value

CIOs look beyond ROI to gauge customer interactions, sales impact, and tech-driven innovation.

No fuzzy goals for Wagner

No fuzzy goals for Wagner
In some cases, IT organizations are applying well-established metrics in new scenarios. The University of Pittsburgh Medical Center has adopted Capability Maturity Model Integration, developed down the street at Carnegie Mellon University's Software Engineering Institute, as well as the institute's Standard CMMI Appraisal Method for Process Improvement, known as Scampi. CMMI isn't new; it's used by defense contractors and manufacturers. The model's use in health care, however, is novel.

CMMI helps companies identify a place to start a project, develop a framework to prioritize actions, and define how improvements will benefit the organization. It also measures the benefits of a process against those realized from similar projects previously undertaken.

Among the medical center's projects that benefited from CMMI is U-Pay, a Web system that lets staff use one program to record payments of various types, such as for patient services, parking, and donations. As components were added to U-Pay, practices embodied in CMMI allowed IT developers to get them right the first time. That saved $20,000 with each release or $300,000 over three years. "Our projects don't fail," says Chris Carmody, director of the software engineering process group within UPMC's Information Services Division.

When the University of Pittsburgh Medical Center adopted CMMI, it benchmarked its performance in developing interfaces between high-volume, data-intensive health and patient management systems against its previous development efforts (see "Development Speed"). UPMC spends more than $220 million a year on IT, including $100 million on capital projects. Competition among UPMC units for that money is stiff, and methodologies and metrics to determine how best to deliver services are indispensable. "There's a lot of consternation every year over who gets what dollars," says Mike Wrobleski, the Information Services Division's finance director. "There's only a limited amount of dollars that can go around, so we really wanted to make sure we got it right."

To be sure, established metrics like CMMI are deeply rooted in many companies and will be for years. Among InformationWeek 500 companies, about a third use the CMMI and Six Sigma methodologies in their daily IT operations, and half use the IT Infrastructure Library, or ITIL, for things such as problem resolution and change management. And CFOs and boards of directors will continue to insist on metrics such as ROI and return on capital, says Ian Campbell, CEO of Nucleus Research. "Return tends to be the driver for all investment decisions," he says.

Yet Campbell agrees that new times call for new approaches. The trend toward business process outsourcing, where work is distributed to contractors who may be based in locations around the world, calls for metrics that let business and technology executives measure the value of every partner, maybe every person, involved in that process. He suggests an outsourced customer-service call center as an example of where that might work. "It can change the way we measure things from a project to an individual," Campbell says.

CIOs must think hard about whether they've got the right tools to track IT's contribution to business objectives. In Optimize magazine's just-completed CIO Effectiveness survey of more than 700 CXOs, CIOs, IT staffers, and line-of-business managers, seven of 10 CXOs say the most important criterion in considering the effectiveness of a CIO is the CIO's ability to support companywide business strategies. Metrics demonstrate that kind of support in hard-to-argue terms. Full results and analysis of the CIO Effectiveness survey will be presented in the December issue of Optimize (InformationWeek's sister publication) and, beginning Dec. 1, at

At United Stationers, CIO Dave Bent wants to start measuring services to the business. He plans to create a "services book" that defines all the services IT provides to the business, using ITIL definitions as the base, and to measure the performance and cost for all those. Under United Stationers' current model, Bent's organization tracks the calls received and resolved by the IT help desk for PC support; the new approach will measure how satisfied employees are with their desktop tools and related services.

"I don't think we're tracking the ultimate customer satisfaction," Bent says. "Could the service be so bad that people don't even call? Have we truly measured the root cause of these calls?" The answers to those questions may require a new tape measure.

-- with Chris Murphy and John Foley

Continue to the sidebar:
Techniques For Measuring IT's Effectiveness

Continue to the blog:
Researchers Measure E-Mail's Potency As Business Tool

Editor's Choice
Joao-Pierre S. Ruth, Senior Writer