06:20 PM
Connect Directly
Repost This

Anti-Spyware Vendors Mad About Consumer Reports Test Methods

Vendors including Microsoft and Sunbelt Software say the consumer magazine's test is bogus because it doesn't take into account how security software detects and removes threats.

Consumer Reports, the independent product review and rating publication, was slammed Friday for using what security experts called "mind-boggling" and "useless" tests of anti-spyware software in its current on-the-stand issue.

"This is beyond anything I've ever seen," said Alex Eckelberry, chief executive of Sunbelt Software, a Clearwater, Fla. security company. "They ran a test that is not a full test of anti-spyware software capability. Consumer Reports scanned for and removed functionality that isn't even real. When I heard what they did, I went 'huh? They did what?' This is just mind-boggling."

For a story on consumer-grade anti-virus and anti-spyware software in the September issue of Consumer Reports, the publication's testers ran various products through the mill. To judge anti-spyware titles, the magazine used the public suite of Spycar scripts published by Intelguardians Network Intelligence. The Spycar site touts the suite as "tools designed to mimic spyware-like behavior, but in a benign form."

Eckelberry took exception with using Spycar in general, and with using only Spycar specifically.

"It's not a serious testing tool," he claimed. "The mantra of any type of security test is that you have to test against real-world scenario. Relevancy is critical."

Spycar fails on those points, he went on. "It does things like install fake registry keys, changes your start page and the like. It is specifically designed to test how well anti-spyware programs block unknown applications, not [how they] scan and remove."

Randy Abrams, for 7 years the man responsible at Microsoft for ensuring that all software it released was malware free, was even more blunt. "I was livid about the testing [Consumer Reports] did. They tested anti-spyware software without ever testing how it detected and removed spyware."

F-Secure's Anti-Spyware and Webroot's Spy Sweeper 4.5 (now superseded by 5.0) tied for first in Based Consumer Reports' tests with matching scores of 89 out of a possible 100. Sunbelt's CounterSpy clocked in at seventh with a score of 70, while Microsoft's free Windows Defender came in dead last with a score of 43.

Eckelberry said it wasn't sour grapes over a low score that lead him to take on Consumer Reports' testing.

"No test is perfect," he said. "But there are certainly degrees [of imperfection]. It should be all about relevancy, but here it's not."

Consumer Reports' September issue was already under fire over its anti-virus testing procedures when Eckelberry raised the flag on anti-spyware. Within days of the issue making the newsstand, McAfee posted an open letter taking the publication to task for hiring a lab to create 5,500 new variants derived from half a dozen malicious code categories.

"Creating new viruses for the purpose of testing and education is generally not considered a good idea," wrote McAfee's Igor Muttik in an entry on the security company's Avert Labs blog. "Viruses can leak and cause real trouble."

1 of 2
Comment  | 
Print  | 
More Insights
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
Twitter Feed
Audio Interviews
Archived Audio Interviews
GE is a leader in combining connected devices and advanced analytics in pursuit of practical goals like less downtime, lower operating costs, and higher throughput. At GIO Power & Water, CIO Jim Fowler is part of the team exploring how to apply these techniques to some of the world's essential infrastructure, from power plants to water treatment systems. Join us, and bring your questions, as we talk about what's ahead.