September 15, 1997

Testing For 2000 By Bruce Caldwell and Andy Patrizio on't look now, but fixing the year 2000 problem is only half the job-maybe less. Testing the fixes could, well, test your patience, resources, and skill. AT&T, for example, estimates that testing its 2000 fixes will eat up as much as 60% of the time and money needed for its total compliance efforts. "We'll test in one fashion or another everything we touch, everything we have that will be impacted, from elevators to application code," says George Brucia, the company's VP for year 2000 effo rts. While companies started their year 2000 compliance projects at different times, and some testing will continue after 2000, everyone faces the same finish date. Meeting that deadline will require more coordination among business units, business partners, software and hardware vendors, and entire industries than currently exists or is even contemplated in many quarters. The need to test year 2000 projects has been overshadowed by the need to raise corporate consciousness about date-field problems and muster the means to fix them. But testing the fixes will involve testing on a scale never seen before. Until now, most testing done as part of routine maintenance and enhancement touched maybe 20% of all code. But year 2000 tests will need to evaluate 80% or more. Worse, there's little help at hand for making testing easier. In fact, tools will reduce the total year 2000 testing effort by just 2%, according to a study performed for General Motors Corp. Also, while replacing older computer systems with new ones that are year 2000 compliant will eliminate some code renovation, it won't erase the need to test the new systems for business functionality and year 2000 compliance. "Very little of the testing effort can be automated," warns Michael Yudkin, VP of corporate systems and architecture at Chase Manhattan Bank in New York. In fact, most automation is done in the remediation phase, which accounts for just 10% of the total effort, he adds. One of the more important tools Chase will use is MainWare Inc.'s HourGlass 2000. It can simulate dates without changing the system clock, "faking it out into thinking it is year 2001," explains Yudkin. Other key tools will provide data routines, change control, version management, and synchronization, he adds. Also, unique testing facilities are required-and they don't come cheap, whether home-built or leased from vendors. Comdisco Inc., for example, charges $7,000 to $100,000 a day, depending on the system configuration, for its year 2000 testing facility . While some companies need to lease test beds for only a half-day, others lease them for six months. Also, nearly everyone plans to begin full integration testing at the same time-in late 1998-making for an unprecedented concentration of effort that is sure to stretch resources to the breaking point. Penalties for failing to thoroughly test year 2000 fixes could be painful, say consultants. They point to possible shareholder lawsuits against corporate officers and directors for business losses resulting from a lack of care. Also, customers could defect to other suppliers, or production could be shut down by systems that fail to work properly. Despite these potential consequences, few companies have squarely faced the issue of year 2000 testing. "Clients underestimate the effort required," says Randy Pelton, director of year 2000 testing at DMR Trecom, an Edison, N.J., unit of Amdahl Corp. Dan Clark, director of application services at Trigon Blue Cross Blue Shield in Richmond, Va., sees it diffe rently. "The magnitude of testing is scaring a lot of folks" in other companies, he says. Clark says some year 2000 managers are taking shortcuts, omitting entire testing procedures to downsize the effort. Good risk assessments can lighten the pre-2000 testing burden by postponing full tests of systems that aren't critical to the survival of the business (see related story " What To Test When You Can't Test Everything "). While every chance to safely lighten the load should be taken, it may seem like bailing out a sinking ship with a teaspoon. The magnitude of the project affects more than just IS staff. At Trigon, the testing burden on the business units was so great that Clark hired temporary end users. He put them through both the training programs designed for regular business-unit employees and year 2000 testing training (see related story " Virginia Insurer Sends In The Testers "). What's involved in testing? Ian Hayes, a co-author of The Year 2000 Software Crisis: Challenge of the Century (Prentice-Hall, 1997) and a principal of Clarity Consulting Inc. in Marblehead, Mass., points to four major components of year 2000 testing: testing management, test data creation, test execution, and test validation. In addition, there are three stages of testing: unit testing as the prioritized applications are remedied; systemwide integration testing after all corporate systems have been remedied; and testing of the data exchanges and transactions between companies and external partners, customers, and suppliers. Testing-unlike assessment and remediation, the initial stages of a year 2000 project-doesn't get easier over time. While some best practices and tools can be developed and shared in testing, as they are in assessment and remediation, "the business knowledge component" essential for testing does not exist outside the business units, DMR's Pelton warns. "Systems are typically tested for new function ality," he adds, "while year 2000 compliance is new functionality across all product lines, and it's difficult to access the businesspeople." Businesspeople are needed as part of the testing process. The IS staff can certify that software code will run properly in the year 2000. But they can't answer whether the software will still perform the business functions it was built to perform. "The businesspeople are the ones who have to certify to that fact," says Larry Winkelman, CIO at Sanwa Business Credit Corp. in Chicago. He estimates that business units can expect a year 2000 testing workload equal to 10% to 20% of the IS group's year 2000 workload. One reason for the huge chore: The number of systems that must work together is awesome. Winkelman's Sanwa unit, for example, is 10 years old, but its year 2000 inventory turned up 215 software components, 95 hardware components, and 115 hardware and software vendors. "We've done other inventories for assessment purposes, says Winkelman, "but not to this extent." Unprecedented Scale One of the biggest year 2000 tests will be done by the 3,500 member firms of the Securities Industry Association. SIA members have conducted tests together before for industrywide changes, such as settling transactions in three days instead of five, but nothing as complicated as year 2000 testing. "This is the largest business technology event that has ever taken place," says Arthur Thomas, chair of the SIA's year 2000 oversight committee, and senior VP and director of global operations services at Merrill Lynch & Co., the $25 billion financial services firm-and SIA member-based in New York. Merrill Lynch has about 220 million lines of mainframe and distributed computing code to remedy and test, at a cost of about $200 million. "It's huge," Thomas says. The SIA's year 2000 testing will involve thousands of brokerages, exchanges, investment banks, settlement processors, international financial transaction networks, and federal and industry regulators. There will be a big test one weekend in the first quarter of 1999, as well as others throughout the rest of the year. The SIA is trying to make sure that participation in the 2000 tests extends beyond the IS group. Thomas' committee was formed earlier this year to make sure the CEOs of every participating company became personally involved. The committee also asked the National Association of Securities Dealers to make year 2000 compliance part of the normal audit examination. But some companies have found that even with massive tests, systems can go wrong. Tec-America Corp., a unit of the Toshiba Group, has already been hit by a lawsuit filed by Produce Palace International, an upscale Warren, Mich., grocer. Produce Palace alleges that its two-year-old point-of-sale system repeatedly crashed because of credit cards with year 2000 expiration dates. The company's lawsuit came even though the credit-card industry has conducted 150,000 field tests of credit cards expiring in year 2000. Part of the problem may be that 150,000 tests represent only a minuscule sample-in all, 21 million merchants accept credit cards. And while the go-ahead has been given on issuance of cards expiring in 2000, credit-card transaction tests were only a part of the year 2000 project at Visa International and other credit-card organizations (see related story " Visa Debits The Vendors "). Visa expects to finish remediation of its internal systems next year, leaving all of 1999 for testing. "Testing is the biggest challenge and key to success," says John McCarthy, VP in charge of the year 2000 project at Visa in Foster City, Calif. "If you mess it up, any of the good work you did in code conversion is worthless." Noxious Networks Even testing itself can be hazardous. "You have to be careful with networks," cautions Debbie Freedman, senior VP of AMR development at the Sabre Group, the Fort Worth, Texas, IT unit of American Airlines' parent, AMR Corp. She says that d uring so-called time-machine tests-in which the system clock is advanced-networks can be brought down and E-mail can be automatically deleted. Also, there are times when a particular piece of hardware can be used for one purpose but not others. Tracking these potential pitfalls is a major task, complicated by the need to re-create environments for testing purposes so that development and production environments are not inadvertently affected. Another sensitive area is end users, says Stephen Greif, a senior programmer at the Johns Hopkins University Applied Physics Laboratory in Laurel, Md. Greif, in charge of the year 2000 team for the lab's business and administration systems, says the year 2000 issue was grasped relatively quickly by executives in the high-technology environment of the lab. But testing can be politically sensitive when, for example, a pet standalone system, such as a database or spreadsheet, needs to be checked for compliance and it's discovered that there's no room on an application s creen for a four-digit year to replace a two-digit year. At Motorola, for example, "there's a lot of code in-house that won't be renovated because the owners say it's already compliant or it will be phased out or whatever," says Chuck Novotny, manager of the year 2000 renovation project at the company's semiconductor products sector in Tempe, Ariz. "We need to do a certain amount of testing to make sure those systems work. You have to test everything, whether you renovate it or not." A lot of tools are available to help with year 2000 testing. Most are the same general-purpose testing tools already in widespread use. These include Computer Associates' CA-Datamacs/II for creation of test databases; Compuware's AbendAid for analyzing the causes of abnormal application program terminations; and IBM's Teleprocessing Network Simulator for large-scale performance and capture-and-playback testing. Date-simulation tools have been developed specifically for year 2000 projects, and these can be useful in both t he assessment and testing stages. These include MainWare's HourGlass 2000, Isogon's TicToc, Prince Software's Simulate2000, and Viasoft's Via/Validate. "Most of the leading firms are using the test tools already available in their current arsenal," says Kazim Isfahani, a research analyst in Giga Information Group's year 2000 relevance service. Isfahani's group is conducting a best-practices study of year 2000 testing at more than two dozen major companies. But methodologies for tackling 2000 testing may be of greater importance than tools. Chase Manhattan, for one, is applying the expertise it gained during its massive integration with Chemical Bank to its year 2000 testing. "There's a high level of similarity between year 2000 testing and the testing we did for the merger," says Brian Robbins, Chase's VP of enterprise IT architecture and its year 2000 manager. Yudkin says the bank put together a project life cycle up front, identified the testing pieces, and then distributed the list to branche s worldwide to create a common set of nomenclature and tasks. Chase used traditional project-assessment practices in establishing a baseline, looking at regression testing, user acceptance testing, and integration. For those without that vast experience, services vendors are preparing methodologies. Consultant Hayes, for example, recently worked with Viasoft, a Phoenix tools vendor, to develop year 2000 testing methodologies for a client, Keane Inc., a Boston software services company with nearly 300 year 2000 projects. Similarly, clients of Computer Horizons Corp. are starting to test year 2000 conversions. The Mountain Lakes, N.J., services firm expects to produce a testing methodology by September, says Arthur Quinlan, its executive VP of solutions. Best practices drawn from the testing experiences of four major clients in different industries will be used, he adds. Other resources needed for year2000 testing include project-management skills to handle complex logistics. It's a process that Al lan Graham, senior VP of operations for continuity services at Comdisco, in Rosemont, Ill., describes as estimating how much time will be needed and when. "That's where the art comes in," he adds. Comdisco ran three shakeout tests, two on mainframes and one on an IBM RS/6000. It found that disaster-recovery managers' skills transfer well to year 2000 tests. One huge challenge to testing year 2000 fixes isn't technical at all. Scheduling the use of a testing facility has gotten difficult. Many facilities have little flexibility in their schedules, particularly during the heaviest period of testing, which is anticipated to be next year. At IBM, for example, the fully staffed multivendor recovery centers are booked through 1999, and already some customers have come for validation tests. "We aren't encouraging customers to change dates because it's hard to juggle things around," says Ronald Beauchamp, year 2000 national solutions executive in Sterling Forest, N.Y., for the business-recovery service s offering of IBM Global Services. Aggravating the scheduling issue: Many 2000 tests depend on integrated products from third-party vendors. "You're held up by the product with the latest date," says AT&T's Brucia. AT&T has set a deadline of March 1998 for certifying all its third-party software for compliance, and for having certified versions of its IBM mainframe operating system, MVS, and all associated software. "But you can't wait to do application testing until all third-party software is compliant, or you'll move the bubble past the horizon," says Brucia. No matter how much consciousness-raising might be done around the issue, without testing, testing again, and testing yet again, year 2000 will indeed be a good time for the legal profession-but a nightmare for executives and boards of directors. It may also be a great time to serve up the heads of year 2000 project managers on platters. Take a deep breath, and start testing. Back to News in Review Send Us Your Feedback Top of the Page

---