OLD-SCHOOL THINKING

Mott's emphasis on benefits to the business and his 80/20 innovation-to-support ratio are just two examples of how business technology executives are relying less on metrics that evaluate IT internally--number of servers per system administrator, for example--and more on those that look at IT's impact on the business. Once-popular measurements such as IT investment as a percentage of company revenue miss the value IT brings to a company, says Gary Scholten, CIO of Principal Financial Group, this year's No. 1 company in the InformationWeek 500 ranking. "That's really an old-school kind of metric," he says. "It's an interesting data point, but not an area of focus for us."

	Madnick investigates: What's real and what's just perception?

Principal doesn't ignore metrics that look internally at IT performance--they're useful in benchmarking against competitors--but it sees greater value in measuring how IT impacts corporate operations. Scholten cites Principal's Work Secure and Retire Secure Web services, which help its clientele of 15 million workers employed by 100,000 small and midsize companies manage and buy retirement, investment, and insurance products. The company tracks demand for its online services compared with offline demand. For example, employee participation in Principal's services is 52% higher through Work Secure, and the "close ratio" is 100% greater.

Such metrics help Principal identify the right level of IT investment. "It's the business success criteria we're looking at, and tying our IT investments that were made in those areas to those business strategies," Scholten says.

Electronics parts distributor Avnet weighs spending on infrastructure and applications against gross profit, which, unlike revenue or expenses, reflects how efficiently the company operates. "It makes sure we keep our IT spend in line with the basic profitability of the organization and doesn't allow IT spend to get out of sync with the amount of money we spend on other operational expenses," CIO Steve Phillips says.

CIOs must strike a balance between measuring performance to squeeze costs and making sure they're spending enough on the right projects to improve the business. Vanguard Group CIO Paul Heller thinks of metrics in something of a Maslow's Hierarchy, a pyramid of six big areas:

• Security and disaster recovery are the foundation, where measurements range from fraud attempts to Web site attacks.

• System availability and responsiveness, with measures like Web site response time.

• Project delivery, including timing and budget.

• Quality, speed, and efficiency in software development, with gauges such as defects allowed into production and lines of code written per developer.

• External client measures, such as usage of new features on the Web site or drop-off rates for new functions.

• Innovation, plotted on a grid with client impact on one axis, operational impact on the other axis, and high, medium, and low as the values.

Heller focuses most of his attention at the bottom of the pyramid, around security, and at the top, on innovation.

EXPERIENCE STILL COUNTS

For an application rollout at its offices in Brazil, storage vendor EMC measured the costs, performance, and quality of outsourcing the work versus doing it in-house. The contractors didn't need to be based in Brazil, but EMC wanted them in or near the same time zone. It commissioned 15 technologists in Mexico to complement 22 of its own managers and employees in Brazil. EMC wanted to see what effect "near-shore" contractors would have on the success of the project. Among the data points analyzed: cost of labor, project budget and schedule, and software defects (see "Staff Mix").

Not every decision goes strictly by the numbers. EMC's IT department still relies on its own employees for project planning and business analysis, areas where it prefers to maintain a direct touch with its primary customers, the company's internal business units. "I don't have the ability to tie a metric to that work," says Ken LeBlanc, EMC's senior director of business operations and portfolio management. "We just know in our gut that is the last area" where outsourcing would be considered, he adds.

Indeed, good numbers are no substitute for hard-earned experience. Metrics can "get in the way of getting customer satisfaction and getting business done," says Bill Kwelty, CIO and a 40-year veteran of fleet leasing firm Automotive Resources International. Kwelty has tried tools that measure how much of the IT staff's time is spent on development and maintenance but found them cumbersome. "The management to oversee all of this didn't seem like time well spent," he says.

Point taken. It's not enough for IT metrics to be relevant; the task of number crunching can't sap productivity.