We were developing the navigation system for a new nuclear submarine and my forecast predicted that 2,000 bugs would have to be found and fixed. The official estimate coming from development was 200--the difference was the cause of the note. Being persistent, stubborn, or just foolish, I continued to track the defect rate. When the project was completed, there were 1,907 errors found and fixed. I knew better than to say, "I told you so," and I stayed in my cubicle.
The next note read, "Come see me." I was nervous but relieved when the big guy asked me, "How did you do that?" The question led to an assignment as a software quality specialist. So I learned (and he learned, too) that when there is serious money and client satisfaction at risk: Accurate Projections = Happy Clients.
So here I am, 17 years later, still preaching and proving that software quality is predictable, so therefore we can set realistic, achievable quality targets.
Now, having worked with a large number of Fortune 500 companies and hundreds of senior managers, I can say that predictability is no longer the challenge. The real challenge today is for companies to make the commitment to long-term process improvement. And like all systemic changes, it is a top-down decision.
When IT management tackles software quality initiatives, not as slogans or fads, but as systematic strategies to improve the performance of delivered products, it's critical to establish reasonable return-on-investment criteria. These criteria should drive toward the heart of business value--increased revenue or decreased expenses. You can't ignore the ever-present triad of better, faster, cheaper--your ROI framework must address these and tie back to the bottom line or the quality program will be abandoned as soon as financial hard times strike.
To do this effectively, one must consider carefully the definition of quality within the context of the business model. A universal standard for quality won't work, and measuring oneself against the wrong standard will frustrate and derail the quality initiative. If a vendor of products for the home PC user tries to define its quality against the exacting criteria of an aircraft flight-controls business, it will never hit the ROI target. The former must focus on qualitative "soft factors" such as customer satisfaction and ease of use. The latter must have "hard factors" to validate near-perfect uptime to ensure no loss of life.
The common thread here is that all businesses must define both hard and soft factors that measure performance and satisfaction for their "products" (either internal or commercial). A clear link to revenue or expense targets must be drawn to these factors. Here's a hypothetical example (a conglomerate of several clients).
Let's say that we build network-management applications for the financial sector and a goal of the company is to leverage a quality-improvement initiative to differentiate our products from the competition and ultimately increase revenue from this product line. To do this, an approach is needed that clearly connects the dots between quality enhancement and increased revenue.
The tactical approach to improve quality has three components. 1) Developing a software-requirements matrix linking all test cases to explicit requirements, 2) formalizing defect tracking earlier in the life cycle, and 3) establishing an independent verification and validation group that's outside the reporting structure of the development team.
Companies that have been effective at putting software quality initiatives into action focus on two questions: what will constitute success, and how do we measure it? They put as much emphasis on these questions as they do on the quality initiative itself.
Here are five essential steps to determining ROI for software quality:
In the case of our network-products example, the business looks at four primary measures that focus on user satisfaction and inherent product reliability. All measures are explicitly linked to alleviating client dissatisfaction with competitive solutions, thus capturing their business and increasing revenue.
One example of a satisfaction measure for this company is the total time to identify a message, flag it, and notify the user. The revenue target could be as simple as 5% annual increase in network management product sales.
Most important, benchmarking is the best way to establish a foundation for ROI that's grounded in hard facts, thus avoiding emotional negotiations as to whether targets were satisfied.
From our example, one of these explicit measures and goals would be "improving the accuracy of message identification from the current benchmark of 0.5 seconds to 0.3 seconds".
With this approach, the success of the software quality initiative for our hypothetical company translates to measurable improvements in performance for users and delivery of a more stable product that meets or exceeds industry benchmarks. In addition, release dates are satisfied with more regularity as cycle times decrease with a reduction in testing. Remember: better, faster, cheaper. The users of these products spread the word when their administrative tools make their lives easier, driving in more business. This is a real, competitive advantage that will propel revenue.
These techniques can be used before, during, and after investments in software quality and should become a systemic component of any "process improvement" initiative. Ultimately, as with most business decisions, the determining factors are mostly financial. They must be based in reality, using appropriate measures that suit your business and industry. A base of quantitative benchmarks and realistic targets must support them. And, they must be linked to increased revenue or decreased expenses. Then IT executives can treat their investment in software quality like any other important business initiative.
Ira Grossman ([email protected]) is founder and president of Emergeon LLC, a professional services and custom software-development firm in Troy, N.Y. Emergeon specializes in metrics-based solutions supporting on-target software delivery for its clients.