InformationWeek

Beta Addicts
These computer professionals can't resist early versions of commercial software. Are they helping you?

By Joseph C. Panettieri Issue date: Oct. 9, 1995

n a previous life, Matt Merrick must have been an exterminator. The VP of information systems at Merrick Printing Co. in Louisville, Ky., Merrick has a knack for tracking down bugs--software bugs, that is. Inside Merrick's desk drawer sit scores of diskettes and CD-ROMs that contain software not yet fit for production computers. Just call Merrick a beta software buff, one of countless computer professionals who are driven to test software before its general release.It may sound nerdy, but in this wired decade, testing beta software has become as fashionable as drinking designer coffee or eating pizza crust-first. It's also a way to one-up colleagues. "There's a community of techies who want to be in on every beta test," says Dwight Davis, editor of Windows Watcher, a newsletter in Redmond, Wash. "They certainly don't want to be one-upped by a buddy down the hall by not having the latest beta." Adds Bruce Brown, editor of BugNet, a Sumas, Wash., newsletter that tracks software quality, "For many, beta testing software is a compulsion." But does beta testing make sense for information systems departments? That depends. Participating in such beta programs represents an equal mix of corporate opportunity a nd risk.

On the one hand, beta testing can help companies understand where technology is heading, which products appear promising, and--equally important--which software isn't worthy of deployment. On the other hand, the beta process has been diluted in recent years, say critics. Beta testing used to be a final spot-check of software that was about to ship. Now, in many cases, it's become a massive bug hunt on products that aren't even close to completion--not the best allocation of an IS department's scarce time.

Beta testing also can bog down an IS department with ominous software bugs, incomplete feature sets, and shoddy product documentation. "Any CIO who says he hasn't had at least one horror story testing software is either lying or has only been in business for six months," says Larry Blevins, senior VP of information technology at the Harris Methodist Healthcare Center in Fort Worth, Texas.

Beta 95
Yet beta testing is burgeoning. Just ask Microsoft, which last spring distri buted test versions of Windows 95 to more than 400,000 users, charging each $30 to participate in what the vendor called the Windows 95 Preview Program. All told, Microsoft estimates more than a million users worked with Windows 95 prior to the operating system's final release on Aug. 24.

Other large-scale beta tests await. In the coming months, beta buffs will have a lengthy list of incomplete, not-ready-for-prime-time products to debug, including test releases of Lotus Development's Notes 4.0 and operating system upgrades from Apple, Microsoft, and IBM.

Hardware makers sometimes use beta testers, too, though less frequently. Network hardware vendors Bay Networks, Cabletron Systems, and Cisco Systems occasionally offer products for beta testing, for example. But such tests are limited because hardware--unlike virtually free copies of software--is so expensive to build.

It's not hard to see how beta testing benefits vendors and individual testers. Vendors get invaluable--and essentially free--hel p in identifying product inconsistencies and stamping out bugs. Beta testers get a thrill from seeing software prior to its release and enjoy hacking products to find their weaknesses. A select few even become technology-industry celebrities. Beta tester Merrick of Merrick Printing (yes, his father founded the company) is regularly quoted in the computer trade press--and even in Microsoft press releases.

Another oft-quoted beta buff is John Miller, a network administrator who has tested a wealth of Novell software for his employer, Healthcare Compare Corp., a provider of health-care services in Downers Grove, Ill. Other beta testers shy away from such publicity but nevertheless win awards for their testing expertise .

Wanted: Experienced Testers
Getting started as a beta tester can be difficult. Software suppliers usually verify an applicant's expertise and intended use of the software. Also, in a Catch-22 situation, vendors want applicants who have alrea dy tested other packages. But, says tester Merrick, "once you're approved and you do a good job, it snowballs into beta testing more and more software."

Just ask Skip McDowell, an information resources engineer at Georgia Power Co. in Atlanta. Microsoft approved McDowell as one of the original 300 testers of Windows 95 mainly because he had previously tested MS-DOS 5.0 and Windows 3.0.

But the benefits of beta testing for IS departments are more difficult to measure. One potential gain: Beta testers often help vendors shape a product's feature set. That gives their company an early view of how a program performs; it also gives an IS department a chance to persuade the vendor to add features that suit its company's specific needs.

McDowell, for instance, once evaluated a network management platform that had poor support for system alarms. He got the vendor to add an option that would forward alarms to a network manager's electronic-mail inbox.

Problems are common as well. Blevins of Harris M ethodist Healthcare Center vividly recalls a four-year-old personal horror story: The center's IS department had for several months examined Elipse, a code generator from Elipse Software Corp., before deploying it--only to discover that the product didn't scale. "It just didn't hold together, so we had to pull the plug," says Blevins.

Other testers have had more disastrous experiences. Boehringer Mannheim Corp. (BMC), a health-care equipment maker in Indianapolis, crashed its Novell NetWare server and knocked several hundred people off line after installing a beta version of a desktop application the company won't identify. "It wasn't a pretty sight," says Peter VanVleet, a systems integration specialist at BMC.

Though such crashes have convinced many companies to formulate policies against running beta software on production computers, once-burned testers don't need any such warning. Miller of Healthcare Compare, for instance, tested NetWare 4.x and the follow-on releases of the network operating sys tem on three servers he set up under his desk. "You've got to take betas slowly and cautiously," Miller says. "Otherwise, you could be out of a job."

Once beta testers have protected their production network, most actually enjoy stumbling across bugs. Many strive to find a "show stopper"--a bug so major it can permanently damage data or delay the product's release. "The best beta testers thrive on finding show stoppers," says Richard Buchannan, a consultant in Peterborough, N.H.

One such tester is Brian Moura, assistant city manager for San Carlos, Calif. Moura has beta-tested operating systems, spreadsheets, word processors, fax gear, and scanning systems, and he's made software testing into a science.

Every time Moura stumbles across a problem, he immediately stops, writes a description of the problem, and files a "bug report" via E-mail to the software's maker. In fact, Moura was one of several Windows 95 testers who discovered that early versions of the systems software quickly ran out of mem ory when multitasking between several 32-bit applications. (Microsoft later corrected the problem.)

But some beta buffs find themselves inundated by vendors pushing evaluation software. Many testers of Windows 95, for example, are receiving calls from Novell, Lotus, and Microsoft asking them to test Windows 95 software suites. There simply aren't enough hours in the day to evaluate all that software and still get their work done. But the true beta buff never says no. "Heck, I've even tested screen savers," concedes Merrick.

Out, Damned Bug!
There's also the sticky issue of getting rid of beta software once the test is over. Many beta applications for Windows can create numerous dynamic link libraries (DLLs), a type of auxiliary file usually stored on the hard disk in the Windows systems directory. It's difficult to identify which DLLs are used by which software; that, in turn, makes it hardto know which DLLs to delete when uninstalling software.

The solution, testers say, is pushing s oftware vendors to include "uninstaller" software with the beta software they ship, or looking on the Internet for uninstaller freeware provided by fellow testers.

Can IS shops keep the software their staff beta tests? It's a common practice, but it's also one that can end in disaster. Most beta software contains time bombs that render the product useless by a specified date. When the bomb is set off, any production systems running on the software could come to a crashing halt.

Beta releases of Windows 95, for instance, will shut down on Dec. 31, 1996, according to Microsoft. That's about the time when many customers are expected to test upgrades to Windows 95 and Windows NT.

Another key question for IS managers is: Are beta-testing employees doing work that should be done by software vendors? "Years ago, a beta product was something with no known severe bugs," says Rich Finkelstein, president of Performance Computing, a consulting firm in Chicago. "The vendor viewed a beta product as something th at was ready to ship as soon as selected customers gave it a go. Now, betas aren't even complete products."

Indeed, vendors such as Microsoft and IBM generally ship several beta releases of software over a period of months, incrementally adding functionality and bug fixes before shipping a final product.

Of course, most software contains minor bugs even in the final release. Just ask purchasers of IBM's OS/2 Warp: They couldn't load the operating system on PCs with IBM's own 486 SLC chip. Or try buyers of Block Financial Software's TaxCut, which can incorrectly calculate the number of dependents on a tax return.

Beta Stew
Proponents of beta testing say such snags are to be expected, given software's complexity. Besides, they add, beta-testing helps the industry. "Software vendors need a stew of PCs to test their software, and the beta process provides that," says BugNet editor Brown.

Help could come from a new development: Beta testing is moving online. CompuServe, for instance, has been home to beta support forums for Windows 95, IBM's OS/2 Warp, and other high-profile software. Generally, participants need permission from the sponsor to access these forums so that it isn't flooded with questions about forthcoming products.

Online beta forums help vendors identify the bugs that users are most concerned about. The forums also foster communication between users and software providers.

Many testers note that prominent Microsoft employees, such as Brad Silverberg, VP of personal systems, answered questions posted on the Windows 95 beta forum himself. "Silverberg and other Microsoft employees provided definitive answers," says Tom Alverson, an engineer at Xetron Corp., a military engineering firm in Cincinnati and a Windows 95 tester.

Online forums also help vendors quickly distribute beta software updates to a large audience. Still, even Microsoft admits that bigger doesn't mean better when it comes to beta testing. "It's a core group of beta testers who do the bulk of the work finding bugs," says Yusuf Mehdi, group product manager for Windows 95 at Microsoft. The core group's name? The Beta Software Buffs, of course.

Comments on this story?

InformationWeek http://techweb.cmp.com/iwk