A Day With The Patch Patrol

The heart of the day's work resided within two XML files that Shavlik's software uses to manage the patch process for its customers. The first XML file is used by HFNetChkPro to assess customers' systems and ferret out which of them are vulnerable to attack. The second XML file instructs HFNetChkPro how to deploy the patch files on customers' systems. "This is what drives our software," Schultze said.

Shavlik's HFNetChk "engine" is licensed and used by other security vendors, including BindView, BMC Software, NetIQ, and Symantec, within their patch-management applications to scan Windows workstations and servers for unpatched and at-risk systems.

First, Schultze prepped the XML file that will be used by HFNetChkPro to scan and assess customer systems for the new vulnerabilities. For about four hours, he prepared the patch-assessment files, painstakingly ensuring that each bit of information--ranging from arcane security-bulletin tracking numbers from security organizations to other minute details about each patch file--is correct, so the software will correctly spot vulnerable systems.

While Schultze worked on Shavlik's patch data files, Karen Helker, the company's quality-assurance manager, began preparing 41 virtual systems that run various Microsoft operating systems and application configurations, to test the XML data files Schultze crafted. "We thoroughly test the detection and the deployment of the patches before making our XML files available to our customers," Helker said.

Helker and her team use the virtual machines to test how well Shavlik's software spots vulnerable systems and deploys Microsoft's patches in each language. For this month's round of patches, that included more than 240 possible patching configurations. Only one of the security bulletins contained a vulnerability in a system for which Shavlik didn't have a pre-built system ready for the test. That was a vulnerability within Microsoft Exchange 2003 running on a Windows 2000 system. "We had that built on Windows 2003, but not Windows 2000, which is the configuration for this particular flaw," Helker said.

By 4 p.m., Schultze passed his XML assessment files to Helker and three other technicians who divided the more than 240 configurations for testing. While the quality-assurance team tested each configuration, Schultze went to work developing the XML file that will be used by Shavlik's customers to deploy the actual Microsoft patches.

The building and testing process continued until late Tuesday evening. At 10:30 p.m., Schultze and Helker held a meeting to determine how well the process had gone for the day and to make sure they were ready to post the patch files to their customers by midnight. The final hours were spent fixing any minor glitches that the quality-assurance process uncovered. "This is the detailed stuff," Schultze said.

It's that kind of diligence that Craig Perkins, bank officer and technology manager at Red River Bank, an independent community bank, says is "extremely important" to him, as the bank's business-technology team navigates through the "labyrinth of corporate host patching."

Red River Bank has been running Shavlik's software for about a year, and Perkins says it has greatly improved the bank's process for patching more than 175 workstations and servers spread throughout seven bank branches. "It's more manageable now," he says. Before running Shavlik's patch-management software, Perkins says, he had occasional sleepless nights, hoping the hacker tools or worms didn't surface before the bank had time to safely patch its systems.

There's not much sleep for the likes of Schultze and Helker on Patch Tuesday. It was just minutes after midnight by the time the process of preparing and testing Shavlik's patch-management software was complete and the files were ready to be published to their customers.

"You missed your same-business-day goal by a few minutes," this reporter prodded Schultze. "No, we didn't," he said. "We're operating on Redmond time."