IT Measures That Matter - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Mobile // Mobile Applications
05:15 PM

IT Measures That Matter

With the right focus, an IT measurement program can return critical information specific to your organization.

An IT measurement program must be built on a foundation of tangible goals that add value. Just as any software development project is positioned for success by aggressively pursuing a clear definition of requirements, the IT measurement program should drive toward consensus in the definition of what's important--and therefore important to measure. This will align the efforts of the measurement program with the stated goals for the IT organization, and the overall enterprise.

Software measurement researcher Victor Basili's Goal, Question, Metric framework defines a tried-and-true measurement model on three levels:

  • Conceptional level: A goal is defined for an object for a variety of reasons, with respect to various models of quality, from various points of view, and relative to a particular environment.

  • Operational level: A set of questions is used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal.

  • Quantitative level: A set of metrics based on the models is associated with every question in order to answer it in a measurable way.

Basili's framework exploits the linkage between what gets measured and what's important (the goals). This linkage is the backbone of a successful IT measurement program.

For example, since Sallie Mae does a significant amount of custom software development, the senior IT leadership team established the goal of improving the quality of its software products. Software quality also was rated the single most important service provided by IT. A number of possible questions that could have been asked to characterize the achievement of this goal were considered:

  • How many defects are generated per thousand lines of code (known as KLOC)?

  • What level of testing effort was required to achieve an acceptable level of quality?

  • What percentage of all software defects that get identified were found by our customers in the user-acceptance testing (UAT) phase of our software development life cycle?

Measure Of Success
Sufficient Horsepower
A part-time commitment won't yield sufficient data
Support From The Top
Needed to resolve turf wars over measurement criteria
Value-Add Goals
Drive toward consensus in the definition of what's important
Customer Surveys
User feedback is invaluable in setting priorities
Don't Imitate Others
Make sure information objectives are specific to your organization

Because defect creation is a function of so many things other than the number of lines of code, KLOC wasn't selected. Business demands for project delivery as well as testing windows that often have fixed durations both compromise how well testing effort can represent software quality. Therefore, testing effort wasn't selected, either. The percentage of all software defects identified in the UAT phase was picked, for four reasons:

  • Defect data was readily available and stored in a trusted central repository.

  • It's the earliest opportunity to capture initial feedback on software quality.

  • It's an objective measure, as these defects are identified primarily by people outside of IT--i.e., by the customers.

  • It hits directly on the desired behavior of finding defects as early in the system development life cycle as possible.

The measure for this goal was intuitive: the number of defects identified during the UAT phase/total number of defects identified (excluding unit testing defects). The performance trend of this goal has been used for the past two years to drive the percentage lower year over year, which supports the goal of finding defects earlier in the development life cycle.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
2 of 4
Comment  | 
Print  | 
More Insights
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Pandemic Responses Make Room for More Data Opportunities
Jessica Davis, Senior Editor, Enterprise Apps,  5/4/2021
10 Things Your Artificial Intelligence Initiative Needs to Succeed
Lisa Morgan, Freelance Writer,  4/20/2021
Transformation, Disruption, and Gender Diversity in Tech
Joao-Pierre S. Ruth, Senior Writer,  5/6/2021
White Papers
Register for InformationWeek Newsletters
Current Issue
Planning Your Digital Transformation Roadmap
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Flash Poll