What Are You Willing To Pay For Quality?

"Whoever generated this quality forecast, tear it up into little pieces and eat it!" This was the note I got from the VP of development in our major software engineering firm. Not the right impression to make in my first job, so I crept back into my cubicle and tried to become invisible.

We were developing the navigation system for a new nuclear submarine and my forecast predicted that 2,000 bugs would have to be found and fixed. The official estimate coming from development was 200--the difference was the cause of the note. Being persistent, stubborn, or just foolish, I continued to track the defect rate. When the project was completed, there were 1,907 errors found and fixed. I knew better than to say, "I told you so," and I stayed in my cubicle.

The next note read, "Come see me." I was nervous but relieved when the big guy asked me, "How did you do that?" The question led to an assignment as a software quality specialist. So I learned (and he learned, too) that when there is serious money and client satisfaction at risk: Accurate Projections = Happy Clients.

So here I am, 17 years later, still preaching and proving that software quality is predictable, so therefore we can set realistic, achievable quality targets.

Now, having worked with a large number of Fortune 500 companies and hundreds of senior managers, I can say that predictability is no longer the challenge. The real challenge today is for companies to make the commitment to long-term process improvement. And like all systemic changes, it is a top-down decision.

When IT management tackles software quality initiatives, not as slogans or fads, but as systematic strategies to improve the performance of delivered products, it's critical to establish reasonable return-on-investment criteria. These criteria should drive toward the heart of business value--increased revenue or decreased expenses. You can't ignore the ever-present triad of better, faster, cheaper--your ROI framework must address these and tie back to the bottom line or the quality program will be abandoned as soon as financial hard times strike.

To do this effectively, one must consider carefully the definition of quality within the context of the business model. A universal standard for quality won't work, and measuring oneself against the wrong standard will frustrate and derail the quality initiative. If a vendor of products for the home PC user tries to define its quality against the exacting criteria of an aircraft flight-controls business, it will never hit the ROI target. The former must focus on qualitative "soft factors" such as customer satisfaction and ease of use. The latter must have "hard factors" to validate near-perfect uptime to ensure no loss of life.

The common thread here is that all businesses must define both hard and soft factors that measure performance and satisfaction for their "products" (either internal or commercial). A clear link to revenue or expense targets must be drawn to these factors. Here's a hypothetical example (a conglomerate of several clients).

Let's say that we build network-management applications for the financial sector and a goal of the company is to leverage a quality-improvement initiative to differentiate our products from the competition and ultimately increase revenue from this product line. To do this, an approach is needed that clearly connects the dots between quality enhancement and increased revenue.

The tactical approach to improve quality has three components. 1) Developing a software-requirements matrix linking all test cases to explicit requirements, 2) formalizing defect tracking earlier in the life cycle, and 3) establishing an independent verification and validation group that's outside the reporting structure of the development team.

Companies that have been effective at putting software quality initiatives into action focus on two questions: what will constitute success, and how do we measure it? They put as much emphasis on these questions as they do on the quality initiative itself.

Here are five essential steps to determining ROI for software quality:

Determine a concise set of measures that relate to quality as defined by the business environment, remembering to allow for both hard and soft factors that affect reliability and customer satisfaction as well as direct measures of revenue or expense goals.

In the case of our network-products example, the business looks at four primary measures that focus on user satisfaction and inherent product reliability. All measures are explicitly linked to alleviating client dissatisfaction with competitive solutions, thus capturing their business and increasing revenue.

One example of a satisfaction measure for this company is the total time to identify a message, flag it, and notify the user. The revenue target could be as simple as 5% annual increase in network management product sales.

Use a benchmark to determine current performance along with realistic quality and revenue and expense goals. This means defining the current quality and revenue levels using the measures described above for a sample of real projects. Realistic targets for each measure should be gauged against company and industry-improvement trends to ensure they're attainable. This can be accomplished by focusing on an efficient set of meaningful measures including schedule, cost, functionality, quality, complexity, and skills.

Most important, benchmarking is the best way to establish a foundation for ROI that's grounded in hard facts, thus avoiding emotional negotiations as to whether targets were satisfied.

Establish short and long-term targets for improvement. In our example, remember we had four quality measures for satisfaction and reliability and one business target (increased revenue). Benchmarking lets us be explicit about these measures and targets.

From our example, one of these explicit measures and goals would be "improving the accuracy of message identification from the current benchmark of 0.5 seconds to 0.3 seconds".

Set up a basic tracking mechanism to capture actual performance for each measure defined above at regular intervals. Refine the measures if necessary and determine if short- and long-term goals are being achieved and the magnitude of the improvement (i.e., critical defects dropped by 10% more than target).

Calculate ROI by isolating the investment made to achieve the quality targets, then comparing it to the increase in revenue or decrease in expenses in that same time period. Any revenue or expense change not in the period before software quality investment can be attributed in whole or in part to the initiative.

With this approach, the success of the software quality initiative for our hypothetical company translates to measurable improvements in performance for users and delivery of a more stable product that meets or exceeds industry benchmarks. In addition, release dates are satisfied with more regularity as cycle times decrease with a reduction in testing. Remember: better, faster, cheaper. The users of these products spread the word when their administrative tools make their lives easier, driving in more business. This is a real, competitive advantage that will propel revenue.

These techniques can be used before, during, and after investments in software quality and should become a systemic component of any "process improvement" initiative. Ultimately, as with most business decisions, the determining factors are mostly financial. They must be based in reality, using appropriate measures that suit your business and industry. A base of quantitative benchmarks and realistic targets must support them. And, they must be linked to increased revenue or decreased expenses. Then IT executives can treat their investment in software quality like any other important business initiative.

Ira Grossman ([email protected]) is founder and president of Emergeon LLC, a professional services and custom software-development firm in Troy, N.Y. Emergeon specializes in metrics-based solutions supporting on-target software delivery for its clients.