All of our efforts to defend against proliferating security threats while complying with endless regulations have one thing in common: They're creating more data for security pros to sift through, a major problem given the near-instant response time that's expected of them.

The promised land? Security information management products that pull timely, relevant information from the logs of various security systems network-wide. These systems detect threats, issue alerts, and generate reports, presenting reams of data in a more digestible and easier-to-act-on format.

The new version of Symantec's SIM appliance includes a Web services API that provides a standardized format for collecting data from intrusion detection and prevention systems, firewalls, patch and asset management systems, and a variety of security applications. Symantec SIM 4.5, starting at $50,000, also improves data compression for storing those files, as well as data correlation and reporting.

The compression feature is essential, especially as companies must preserve their data longer, says Adam Gray, CTO of IT services firm Novacoast. The company has been using a prerelease of Symantec SIM 4.5 since November to aggregate and evaluate its daily data load, which varies from a couple of hundred megabytes to 2 Gbytes.

Novacoast archives security event data for 90 days. With version 4.5, it puts its archive into a flat file, compresses the file, and stores it in a direct-attached storage device. Without the improved compression, should Novacoast need to archive events for years, "that would just be a disaster for us, although it would make the disk manufacturers happy," Gray says. He also likes the additional report templates available in the 4.5 version, which he finds are easier to read.

Ideally, SIM systems bring together event records, prioritize incidents, separate real security violations from false alarms, and aggregate security events from different locations, devices, and manufacturers, IDC analysts Charles Kolodgy and Rose Ryan wrote in a December report. Symantec, now fourth in a market that IDC pegs at $478 million and growing 25% a year, hopes the improvements to its appliance will help it catch market leaders ArcSight, NetForensics, and Network Intelligence (for which EMC paid $175 million in September). Last month, ArcSight increased storage capacity and compression on its Logger appliance.

SIM systems are too costly and complex for most small businesses, which instead collect reports from different security logs and do the threat correlation themselves. But that correlation is getting more difficult as new mandates, such as the amended Federal Rules of Civil Procedure, require businesses to retain electronic records longer in case they're needed in court. "The party line in the security community is to log everything, just in case," says William Bell, director of security at CWIE Holding.

Still, while the centralization of security info is appealing, Bell says he's more trusting of specialized data collection and reporting tools, such as Cisco's NetFlow log analyzer for network traffic. "Jack-of-all-trades" SIM systems aren't as effective, he says.

To broaden their appeal, Symantec and rivals must overcome such notions. If the vendors can deliver, they'll put managers in a better position to see, and act on, tomorrow's security threats today.