InformationWeek

f ever there were an IT blame game that appears truly unwinnable, squabbling over defective software is it. Customers complain that buggy code from suppliers saddles them with system crashes, time wasted devising workarounds, and expensive IT projects in which products often don't work quite as advertised. Software vendors, under pressure to stay ahead of competitors, say customers buy upgrades for enhanced features, not improved quality, and contend that IT users cause many of their own headaches by heavily customizing applications.

Software buyers, facing quarterly financial pressures, have become more willing to hold vendors publicly responsible for problems tied to software, as Nike Inc. did earlier this year with supply-chain management software company i2 Technologies Inc. In response, some of the largest software vendors are promising to crack down on low-quality software, waiting longer to release products to reduce defects, injecting customer scenarios into testing methods, and investing in analytical tools that can squash bugs. That's a litany that sounds familiar to experienced software customers--will they bite this time?

Microsoft says it has changed testing procedures for its E-business server products to reflect customer workloads, in some cases turning documentation of users' daily procedures into stress tests for its software. Oracle has responded to complaints of poor quality in its Oracle 11i E-business software by discouraging customers from extensively customizing the application. Last month, the vendor introduced a series of modular, fixed-price, customer-relationship management applications designed to get customers running in 90 days, with the hope of cutting down on installation woes. IBM, meanwhile, plans to spend as much as a quarter of its $2 billion annual research-and-development budget on a companywide effort to develop systems that recover more automatically from failures, distribute work to idle computers, and manage software upgrades.

"Everything you do in software, you rub up against resources, time, and market pressure," says Paul Flessner, the senior VP in charge of Microsoft's enterprise server business. "Quality isn't a variable. No one wants to make a crummy product."

Despite good intentions, defective code remains the hobgoblin of the software industry, accounting for as much as 45% of computer-system downtime and costing U.S. companies about $100 billion last year in lost productivity and repairs, says the Standish Group, a market research firm. That doesn't include the cost of losing angry customers. Because IT shops write applications that rely on packaged infrastructure software, bad code can wreak havoc on custom apps as well.

"Absolutely, the commercial off-the-shelf software products we purchase don't have the quality we need," says Alan Willett, a software process engineer at Xerox Corp., where badly designed software has resulted in the grindingly slow performance of a multimillion-dollar order-entry system designed to support 1,000 salespeople concurrently, as well as lost productivity of engineers designing workarounds.

But Xerox's IT department is fighting against poor-quality software where it can--within its own ranks. Last fall, it trained five software engineers for more than 120 hours each, at a cost of about $45,000, on bug-stomping methods devised by the Software Engineering Institute, a federally funded research-and-development center operated by Carnegie Mellon University. The process helps programmers catch errors while they're writing code, instead of when IT tests it. That helped Xerox save about $200,000 last fall on a $400,000, 6,800-line set of utilities and transaction-logging services the company hopes to leverage across new Java applications it's writing. Not bad.

Willett says programmers fixed defects in five minutes while writing code, compared with the 23 hours of development time it typically takes Xerox to fix bugs during testing. This year, the Rochester, N.Y., copier and document-management company plans to apply the methodology to a field-service support system that employs 200 IT people and has been hampered by cost overruns.

Just how bad is bad software? Definitions vary, but experts say it takes only three or four defects per 1,000 lines of code to make a program perform poorly. Factor in that most programmers inject about one error for every 10 lines of code they write, multiply that by the millions of lines of code in many commercial products, then figure it costs software vendors at least half their development budgets to fix errors while testing. Get the picture?

"Most software people will tell you they care about quality," says Watts Humphrey, a fellow at the Software Engineering Institute who spent 27 years working on programming quality at IBM. "The problem is the management system [for developers], the training, and the techniques being used are all designed not to do quality work." That is, most vendors like to bang out code, then fix it during the test cycle. Sooner or later, the tests have to cease, and the product has to ship.

The results can make managers dyspeptic. Nike blamed a sales shortfall of as much as $100 million for its third quarter ended Feb. 28 on a botched implementation of supply-chain management software from i2. The athletic shoe and clothing maker says it double-booked orders, producing millions of unwanted shoes, and fell short of supply of popular models after an i2 rollout that began last summer went bad. I2 says Nike pushed the $400 million system into production too quickly and went live with too many suppliers and distributors at once.

In 1999, Hershey Food Corp. went live after 2-1/2 years of development with a $112 million order-taking and distribution system that combined software from Manugistics Group, SAP, and Siebel Systems and touched nearly every facet of its operations. SAP systems normally take several years to fully install. When Hershey's system started delaying orders by as much as a week, the company missed crucial Halloween candy deliveries to retailers, forcing big chains such as Wal-Mart and Kmart to stock up on competitors' products. Hershey estimates the problems cost it $120 million in sales that year.

A few months earlier, eBay Inc. suffered a 22-hour outage of its E-commerce Web site, traced to a bug in Sun Microsystems' Solaris operating system that corrupted information in an Oracle database. Sun CEO Scott McNealy acknowledged a known bug caused the problem.

How did IT get into this mess in the first place? Part of the problem was the rise of client-server systems in the 1980s, which exposed businesses to the quick-and-dirty PC culture. In the '60s and '70s, companies buying mainframes could count on dependable code and predictable hardware and software from vendors such as IBM, Burroughs, and Sperry. But the PC industry, built on high volume, low prices, and frequent upgrades, made hash of a lot of that quality control. Just ask companies such as Oracle and Sybase Inc., which nearly lost their franchises after wrestling with quality problems in the past decade--or Ashton-Tate, which did.

Increasingly complex IT environments don't make matters easier. Nearly everyone who makes or uses commercial software says it's impossible to test for all possible combinations of products that customers create. "The art of software has become more the art of putting together pre-existing stuff, rather than writing it all anew," says Anders Hejlsberg, a distinguished engineer at Microsoft who wrote the company's new C# language as well as Borland's Delphi and Turbo Pascal development tools. "We're constantly fighting with this exponential expectation of more and more."

Compounding the problem is that the rise of E-business and the emergence of computing paradigms, such as Microsoft.Net, that encourage companies to build more distributed apps, mean software problems can quickly affect large numbers of users. "If we end up delivering faulty data, we frustrate customers, and they lose confidence in our capability," says Jack Look, a senior VP at Beta Systems, a Milwaukee subsidiary of Thomson Financial that executes trades and tracks data for brokerages. "The customer doesn't want to hear that there was a problem with the software." Look says problems with data-integration software from Informatica Corp. thwarted his attempts to develop more functions on Informatica's product. Informatica says it has shored up quality control and performance testing in the second release of its flagship product, PowerCenter, just entering the market now.

Oracle has grown so weary of customers that run into problems with its application suite that CEO Larry Ellison spent nearly his entire keynote address at the company's recent AppsWorld conference in New Orleans discouraging customization. "If you start heavily modifying our software and we come up with a new release a year from now, you can't take it," Ellison told users. "You call us on the phone with a problem, we really don't understand the problem, because a lot of that system you've written--not us."

Harsh words. So where do IT managers go from here? Despite their best efforts, developers will inevitably write faulty code. And despite Humphrey's exhortations, it seems unlikely that many vendors will incur the cost of changing time-honored programming methods, which rely on admittedly uncomprehensive tests to fix errors. That means enough bugs will slip through to make software at best annoying, and at worst extremely costly, to use. Perhaps by spending more time measuring quality and approaching projects with a bit more caution, IT shops can tame some unruly software.

BV Solutions Group Inc., an Overland Park, Kan., IT services subsidiary of engineering company Black & Veatch, runs a small software "clearinghouse" in its IT shop to make sure vendors' apps don't conflict with existing software before they're rolled out. The result? A big drop in help-desk calls. "We're probably tying up three or four people," says quality assurance manager Kevin Smith. "But a problem isn't go-ing to end up in the lap of 1,000 users Monday morning."