How do you know when a software-quality problem is hurting the overall business? For Greenleaf Nursery Co., the answer was in the azaleas. A Home Depot or Wal-Mart store might order 100 azaleas from Greenleaf, then decide it needed only 50 and reduce its order online. Right on cue, a Greenleaf delivery truck would show up at the store--with 150 plants.

Offering customers perishable products they don't want isn't good for relationships or the bottom line, but Greenleaf's woes didn't end there. The Park Hill, Okla., company, which sells almost $60 million worth of flowers, shrubs, and trees each year, says faulty inventory reports forced it to dispatch employees every day to re-count plants in the nearly two-square miles of fields in which they grow. Greenleaf claims its J.D. Edwards & Co. OneWorld software "hadn't been adequately tested and debugged" and was responsible for the inaccurate reports and other glitches that led to lost sales, spoiled inventory, and labor overtime costs.

J.D. Edwards won't discuss Greenleaf's project directly, but the vendor defends the quality of OneWorld and says companies' IT infrastructures sometimes can't support projects as complex as enterprise resource planning. "It's the most complicated software made, period. And installation is very much akin to rewiring a nervous system," a spokesman says. Greenleaf has filed a $40 million civil suit that will be heard in arbitration.

"We did lose some customers," says Reece Morrel Jr., an attorney for Greenleaf, which has replaced OneWorld with a customized system based on the Linux operating system and Oracle software. "We just hope to demonstrate in the future that we've got a handle on orders now and win that business back."

Most companies feel at least some of Greenleaf's pain: Of 800 business-technology managers responding to an InformationWeek Research survey last month, 97% report problems with software bugs and flaws in the past year, and nine out of 10 report higher costs, lost revenue, or both as a result. While most disputes don't end up in court, there's a growing consensus among business-technology managers that they will no longer tolerate poor software quality. Companies have become so dependent on software that any failures carry the threat of irreparable harm to their businesses--especially if systems are supposed to operate around the clock.

Where does fault lie: with the software or the customer? Software quality is a complicated issue, and in cases such as Greenleaf and J.D. Edwards, it can take a third party to decide. But vendors are starting to get the message that they need to be more accountable for quality in all its forms, from reliability to security to performance. Oracle, for instance, touts its efforts to make quality a larger part of its development efforts. And there's a school of thought among some in the IT industry that quality is more than making sure code is free of programming bugs or security holes. "Any time software doesn't behave the way we want it to behave, it's a defect," says Amitabh Srivastava, a distinguished engineer with Microsoft Research. Customers would say the same thing.

Microsoft chairman and chief software architect Bill Gates says the industry's future hinges on improving quality. He'd like companies to become even more dependent on software, using emerging standards and technology known as Web services to interconnect every interaction until they're completely digital businesses. That won't happen unless quality improves. "It's critical to what we're asking customers to do in the future," Gates said in an interview last week. "To ask people to do that, they need to view it as a very reliable system." In a January letter to the company's employees, Gates stressed that quality--what he called "trustworthy computing"--had to come before anything else. That means a future Windows version, code-named Longhorn, will be delayed at least four months and won't include some hoped-for features.

Microsoft users may not mind waiting if the delay means better software. The InformationWeek Research survey found that 62% of respondents say the software industry is doing a bad job of producing bug-free software. At most businesses, software patches and bug fixes are a normal part of everyday life: 62% of respondents do periodic updates of patches and fixes, while 24% download them automatically. Only slightly more than a third consider their most-strategic vendors' products trustworthy, and about half consider them somewhat trustworthy.

The momentum behind the software-quality movement became evident last week, when three professors from Pittsburgh's Carnegie Mellon University launched the Sustainable Computing Consortium. Academic consortiums rarely suggest fast and dramatic change, but this one already boasts a roster of companies that have forked over $25,000 in annual dues, including American International Group, CMP Media (InformationWeek's parent company), Mellon Financial, Merck, and Pfizer, plus vendors such as Microsoft and Oracle. The consortium, which also will tap into a $27 million grant from NASA for software research, plans to develop tools that measure software quality, disseminate best practices among members, and research the impact of software quality on the economy (see story, "Consortium Looks For A Long-Term Solution").

USAA, a financial-services company with $60 billion in assets, shows just how intolerant executives have become of software failure. CEO Robert Davis two years ago issued a directive that the company offer 100% availability for live money transactions. If a USAA customer wants to withdraw money, sell stock, or apply for a mortgage, an out-of-commission software system must never block the road. "It's a very tall order," says Steve Yates, president and CEO of USAA Information Technology Co., which operates as a separate company and serves the technology needs of USAA's business units. That's why, on any given day, six well-paid consultants roam the company's San Antonio campus, working with IT to improve the way in which its 1,500 programmers develop software.

USAA is among a growing number of companies that plan to tackle quality problems with the custom software they develop in-house by improving their development processes, and it shows how academic efforts such as Carnegie Mellon's can help companies in the real world. USAA uses the Capability Maturity Model assessment, a five-step standard for quality-development processes created years ago by another Carnegie Mellon organization, the Software Engineering Institute. Assessment is tough at every level--only about 24% of organizations implementing the Capability Maturity Model have made it to level 3, and only about 6% have achieved level 5. The consultants USAA hired and another 10 employees are working full-time to get USAA IT to level 2. "It adds up to a lot of time, and time is money," says Yates, who estimates the company will spend more than $2 million this year on its Capability Maturity Model efforts. "You can't read a book on the subject and be ready to go." USAA IT wants to reach level 3 within five years, which requires the clear definition of all software-development processes and strong feedback on software bugs and other quality concerns.

That still leaves Yates struggling with his biggest concern: lack of control over the quality of the systems USAA's business partners use. Say a USAA customer is trying to close on a mortgage, but USAA can't gain access to a government system that contains information on property taxes. The government's software might be faulty, but the customer holds USAA responsible. Yates' only real ammunition in such cases are service-level agreements between USAA and its partners. What he'd really like is to see software quality become a rallying point for all the industries and agencies he deals with.

At Cleveland's KeyBank, software quality wasn't a constant concern back when customers did their banking with live tellers during regular business hours. But the Internet has changed all that. The bank, which operates in 13 states and has $81 billion in assets, is working to ensure that customers have constant access to their accounts and that their account information is always accurate.

It's not doing badly. KeyBank's IT department regularly achieves availability and accuracy of more than 99% for its systems. But that means there could be times when customers can't access their accounts and occasions when transactions go wrong. "Our customers expect their bank to be accurate and take care of their money. That's non-negotiable," says Bob Dutile, executive VP in charge of technology architecture services. "We need to reduce the effort it takes to make sure everything works, and we need to find fewer problems."

KeyBank uses the Capability Maturity Model as a yardstick but doesn't pursue formal Software Engineering Institute assessments. It's joining the Sustainable Computing Consortium in hopes it will get access to best practices and better results from vendors. The Capability Maturity Model and other such standards measure only processes, not the end product. That's the goal of the Sustainable Computing Consortium. "Our objective is to directly measure the quality of code itself," says Carnegie Mellon professor William Guttman, who started the consortium with associate professor Ashish Arora and Bill Scherlis, principal research scientist at the university's school of computer science. "What we're trying to do doesn't exist today."

To get better quality from vendors, KeyBank's Dutile holds in-person reviews at least once a year and issues quarterly performance reviews based on a scorecard system. The bank also distributes awards for the best vendor and most-improved vendor, which recipients can use in their marketing materials.

There are tools available from a smattering of vendors for testing both commercial and custom-built software for bugs and flaws. Amazon.com Inc. uses software from Geodesic Systems Inc. to diagnose software and detect problems before they occur. "Lack of quality is visible to our customers," says Mike Carr, the online retailer's software development manager. But InformationWeek Research shows many companies aren't satisfied with testing tools, either. Only 5% find tools for testing off-the-shelf software to be adequate, 40% find them somewhat adequate, and 27% say they're inadequate. The rankings for tools that test custom-built applications were 15%, 49%, and 16%, respectively. And in both cases, nearly 30% of respondents don't use such tools.

The survey also revealed mixed results for quality-assurance testing. Just over half of respondents prepare and follow a formal quality-assurance test plan; however, 21% prepare a formal plan but may not monitor results. Business-technology managers say it's a trade-off between quality and time. They have worked at, or still work at, businesses where the quality-assurance plan is sometimes thrown out if the business demands that the software be operating as soon as possible.

Software companies can pursue development-quality standards such as the Capability Maturity Model, and a few, such as Computer Associates, J.D. Edwards, and Information Builders, tout their broader quality standards, but it's something vendors haven't put a high priority on, largely because of the time pressure they face to get products to market.

Quality champions say the speed-to-market obsession must stop. Allan Woods, CIO and vice chairman at Mellon Financial Corp., says the software industry needs to reverse its practice of shipping products first and dealing with flaws later. "Typically it gets back to the question of, 'Why wasn't this issue dealt with when the software was developed?'" he says. Mellon Financial has spent several years working on its Capability Maturity Model certification for internally developed software, and it sees the hard work paying off. Two of Mellon Financial's IT-development groups are at level 3; the bank has seen post-release software defects drop by 50 and reductions in project cycle times ranging from 20% to 50% in those sectors.

Companies such as Mellon Financial think of software quality as a people issue. "If you don't have a world-class organization, the job is impossible," Woods says. Mellon has worked hard to reduce attrition among engineers so it can maintain its knowledge base and the continuity of its software-quality efforts. This isn't the time to scrimp on incentives such as performance-based pay, Woods says.

First Data Corp., a credit-card processor in Omaha, Neb., gets representatives from its top 12 clients, including Visa and MasterCard, involved in the quality-testing process. "They come in and talk to us about how we introduce change and how we can make sure change doesn't impact them adversely," says senior VP of systems and programming Dave McCann. Still, quality has occasionally been an issue for the company, and code change is often the culprit. First Data hopes to reduce quality problems by reducing change. Next month, it plans to roll out a rules-based system that will let its bank and credit-card customers change their product offerings without altering any software coding.

Academic institutions also need to do a better job of teaching quality to budding engineers and developers, says John Dodson, manager of software engineering at a Lockheed Martin division in Newington, Va. "Most of the college folks I get have a real good background in building software and a horrible background in testing it," he says. As a defense contractor, Lockheed Martin can't tolerate shoddy processes and workmanship. Its focus on consistent processes and methodologies in recent years has helped many of its development organizations reach level 4 or 5 of the Capability Maturity Model.

Of course, not all businesses are as sensitive to quality issues as Lockheed Martin, and software doesn't always have to be perfect. One of the Sustainable Computing Consortium's goals is to help companies decide when software needs to be rock solid and when it can simply be adequate. The consortium also plans to research the economic impact of software quality: Is there a point at which focusing on quality could stymie the economic advancement of the software industry or those companies developing software for their own use?

That's a concern of George Rimnac, chief technology officer at industrial-supply distributor W.W. Grainger Inc. in Lake Forest, Ill. He says an IT department can end up building systems that run flawlessly but don't do what the company needs. Rimnac's IT group tracks disruptions in business activity and looks for whether IT systems caused the problem. "The point of software quality isn't to have quality software," he says. "It's to have uninterrupted business support." That's a goal everyone in the industry ought to be able to live with.

-- With Chris Murphy

Photo of William Guttman, Ashish Arora and Bill Scherlis by Bill Cramer.

Photo of Dave McCann by Paula Friedland.