Anti-Spyware Vendors Mad About Consumer Reports Test Methods - InformationWeek
06:20 PM

Anti-Spyware Vendors Mad About Consumer Reports Test Methods

Vendors including Microsoft and Sunbelt Software say the consumer magazine's test is bogus because it doesn't take into account how security software detects and removes threats.

Consumer Reports, the independent product review and rating publication, was slammed Friday for using what security experts called "mind-boggling" and "useless" tests of anti-spyware software in its current on-the-stand issue.

"This is beyond anything I've ever seen," said Alex Eckelberry, chief executive of Sunbelt Software, a Clearwater, Fla. security company. "They ran a test that is not a full test of anti-spyware software capability. Consumer Reports scanned for and removed functionality that isn't even real. When I heard what they did, I went 'huh? They did what?' This is just mind-boggling."

For a story on consumer-grade anti-virus and anti-spyware software in the September issue of Consumer Reports, the publication's testers ran various products through the mill. To judge anti-spyware titles, the magazine used the public suite of Spycar scripts published by Intelguardians Network Intelligence. The Spycar site touts the suite as "tools designed to mimic spyware-like behavior, but in a benign form."

Eckelberry took exception with using Spycar in general, and with using only Spycar specifically.

"It's not a serious testing tool," he claimed. "The mantra of any type of security test is that you have to test against real-world scenario. Relevancy is critical."

Spycar fails on those points, he went on. "It does things like install fake registry keys, changes your start page and the like. It is specifically designed to test how well anti-spyware programs block unknown applications, not [how they] scan and remove."

Randy Abrams, for 7 years the man responsible at Microsoft for ensuring that all software it released was malware free, was even more blunt. "I was livid about the testing [Consumer Reports] did. They tested anti-spyware software without ever testing how it detected and removed spyware."

F-Secure's Anti-Spyware and Webroot's Spy Sweeper 4.5 (now superseded by 5.0) tied for first in Based Consumer Reports' tests with matching scores of 89 out of a possible 100. Sunbelt's CounterSpy clocked in at seventh with a score of 70, while Microsoft's free Windows Defender came in dead last with a score of 43.

Eckelberry said it wasn't sour grapes over a low score that lead him to take on Consumer Reports' testing.

"No test is perfect," he said. "But there are certainly degrees [of imperfection]. It should be all about relevancy, but here it's not."

Consumer Reports' September issue was already under fire over its anti-virus testing procedures when Eckelberry raised the flag on anti-spyware. Within days of the issue making the newsstand, McAfee posted an open letter taking the publication to task for hiring a lab to create 5,500 new variants derived from half a dozen malicious code categories.

"Creating new viruses for the purpose of testing and education is generally not considered a good idea," wrote McAfee's Igor Muttik in an entry on the security company's Avert Labs blog. "Viruses can leak and cause real trouble."

1 of 2
Comment  | 
Print  | 
More Insights
Oldest First  |  Newest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of the Cloud Report
As the use of public cloud becomes a given, IT leaders must navigate the transition and advocate for management tools or architectures that allow them to realize the benefits they seek. Download this report to explore the issues and how to best leverage the cloud moving forward.
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on for the week of November 6, 2016. We'll be talking with the editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll