Get Ready To Patch

In the first three weeks it's been available, businesses have downloaded more than 1 million copies of Microsoft's Service Pack 2 for Windows XP, and consumers have downloaded many more. It's merely a start in what's shaping up to be the most far-reaching and complex software patch ever attempted. Over the next three months, Microsoft's goal is to push SP2 out to more than 100 million PCs. Few expect it to be easy. Applications already are breaking as software vendors and systems administrators test the security-packed Windows update before rolling it out to users. Microsoft has identified about 50 applications that are incompatible with SP2, and company officials admit that many custom applications are likely to encounter glitches, too. Last week, Microsoft released a 100-page technical document that describes how companies should assess applications for compatibility with SP2 and what they should do when things don't work.

Microsoft's mother of all patches is just the latest in what's become a familiar and frustrating industrywide exercise, as software companies and their customers race to stay ahead of the worms and other attacks that seek to take advantage of newly discovered vulnerabilities in operating systems and applications. "You have to take this stuff seriously. You can't let your guard down for a second," says Michael Kamens, global network and security manager with Thermo Electron Corp., which has tested SP2 but hasn't determined a rollout schedule for its several thousand Windows XP machines.

For many companies, patching has been akin to software triage, with IT personnel dropping what they're doing every time a critical security bulletin rings the alarm. A growing number of companies, however, are putting people, processes, and tools in place to bring greater efficiency and control to that ad hoc way of doing things. And technology vendors are making some much-needed changes, too.

Oracle has revealed that it will begin releasing its software patches on a once-a-month schedule, so customers can better plan for them. "We believe a single patch encompassing multiple fixes on a predictable schedule better meets the needs of our customers," Oracle said in a written statement. Oracle also indicated that a security fix would be issued shortly for vulnerabilities that have been discovered in its products but declined to comment further on the pending fix or its revised patch strategy.

Microsoft began issuing monthly patches last October, and Computer Associates and SAP have been on regular schedules even longer. SAP uses its Support Portal to make updates available, including specialized patches for customers who may need help reconciling SAP applications with third-party products. CA delivers patches once a quarter, but it moves faster when necessary. "When I sit down with customers, I seldom get to bring up the issue--it's usually one of the top things they mention," says Sam Curry, VP of CA's e-Trust security-management unit.

Jim Burdiss, VP and CIO of Smurfit-Stone Container Corp., likes the trend toward scheduled patches. "The end game is to get away from fire drills as much as possible," he says. "When those patches happen randomly, you force IT to go into a reactive mode." The randomness of ad hoc patches makes resource and budget planning difficult, he says.

Oracle's policy change and product improvements from Microsoft, including new features in Systems Management Server 2003 that automate aspects of Windows patch management, are steps in the right direction. But challenges remain. The Yankee Group consulting firm estimates that a company with more than 500 PCs spends up to 120 staff hours testing and installing every patch. "The issue is, companies have to test and test before deploying a patch," says Yankee Group senior analyst Eric Ogren.

At the Arkansas Army National Guard, two people work full time patching about 50 Windows servers and 1,500 PCs. "That seems excessive," says senior network manager Lynn Melton. "It's frustrating." The military unit uses several tools to deploy patches, including St. Bernard Software's UpdateExpert, Lieberman Software's User Manager Pro, and Cisco Systems' CiscoWorks. Melton tried an earlier version of Microsoft's Systems Management Server but it required too much effort, he says. He's interested in the vendor's Windows Update Services patch-management system, which promises to let customers handle patches for more products than Windows, including SQL Server and Exchange. But it won't be ready until the first half of next year. "If we could use one tool to do more than one thing, that would be helpful," Melton says. Thermo Electron uses Microsoft's Software Update Services 1.0 tool (the predecessor to Windows Update Services) for patching at its headquarters, but remote locations continue to handle the job locally, so it's a challenge to get everything done quickly. "The problem is, you need a dedicated full-time person to write scripts and push the patches out there," security manager Kamens says. The company is deploying Systems Management Server 2003 to help, but at an estimated total cost of about $1 million, it won't be cheap. Even after predeployment testing, Kamens says, patches too often "break things." But it's something that has to be done--the risks of unpatched systems include worms and other threats, the data vulnerabilities and system snags associated with such threats, and potential liability, lost productivity, and other costs related to any security breaches. Thermo Electron's IT staff rolls out software updates to 800 servers once a month on a Sunday morning to minimize system downtime.

Companies of all sizes are grappling with the issue. Ajacs Die Sales Corp., a small distributor of tool-and-die components, has only VP of IT Steve Wierenga to patch its 22 PCs and four servers. "We have it under control," says Wierenga, who evaluates Microsoft's patches himself each month. "We're small enough that we can address an issue with a patch in short order if it causes a problem." At the other end of the spectrum, software vendor SupportSoft Inc. says one of its customers, a bank with 50,000 PCs, will have dozens of technicians testing the SP2 patch over several months.

Stolt Sea Farm, a seafood company, takes a no-frills approach. The company's IT environment consists of 550 thin-client terminals and 50 Windows servers spread among locations in about a dozen countries. Because there are no desktop PCs to support and most of its software comes from Microsoft, the company's small IT staff is able to install patches within 24 hours--and it does so without any testing. "I would say we are very efficient," says systems administrator Terje Sorgjerd.

CIO Burdiss of Smurfit-Stone Container believes businesses need to master the nuts and bolts of patch management to focus IT resources on what really matters: delivering increased business value. "Before you can do governance and develop the value of IT to the business and all of the things we're trying to aspire to, you have to have some credibility," he says. "In my mind, the lights-on stuff has to work every time, and these patches can be counter to that."

The good news is that companies generally seem better prepared to deal with patches today than a year ago, using patch-management products from specialists such as PatchLink Corp. and Shavlik Technologies LLC and new capabilities from their primary software suppliers. For example, PeopleSoft Inc., which issues patches quarterly, has cut the number of manual steps required to find, download, and install patches and software updates from 49 to seven.

Better defined internal procedures at user companies are helping, too. As a result, the Yankee Group estimates costs have dropped to about $150 per patch for each PC, from about $250 last year. Companies are "better at it than they were 12 months ago," says Michael Cherry, an analyst with Directions On Microsoft. "But it still requires a considerable allocation of resources."

That will be especially true with SP2, which, at a minimum of 75 Mbytes per machine, promises to clog networks if not managed carefully. And once it's installed on PCs, help-desk administrators could see a spike in support calls as users grapple with nuances in the way Microsoft's Internet Explorer browser works with SP2 and other security-related changes. "It's going to cause as many problems as it fixes," predicts Simon King, SupportSoft's director of product marketing for enterprise solutions. "It's going to be a huge undertaking."

Microsoft group product manager Barry Goffe says the company is doing everything it can to help. In addition to the 100-page applications-compatibility document, it has already released a 200-page technical overview of SP2, a Solution Accelerator that provides guidance on how to load Windows XP SP2 onto a computer, and other documentation. Over the next few months, Microsoft plans to deliver the beta version of an applications-compatibility toolkit for SP2, which will automate some manual processes. And next year, improved patch management in the form of Windows Update Services should arrive.

It makes for quite a patch. The next few months will tell just how much companies have really improved at managing it all.

-- With Charles Babcock and Beth Bacheldor