Consumer Reports, the independent product review and rating publication, was slammed Friday for using what security experts called "mind-boggling" and "useless" tests of anti-spyware software in its current on-the-stand issue.
"This is beyond anything I've ever seen," said Alex Eckelberry, chief executive of Sunbelt Software, a Clearwater, Fla. security company. "They ran a test that is not a full test of anti-spyware software capability. Consumer Reports scanned for and removed functionality that isn't even real. When I heard what they did, I went 'huh? They did what?' This is just mind-boggling."
For a story on consumer-grade anti-virus and anti-spyware software in the September issue of Consumer Reports, the publication's testers ran various products through the mill. To judge anti-spyware titles, the magazine used the public suite of Spycar scripts published by Intelguardians Network Intelligence. The Spycar site touts the suite as "tools designed to mimic spyware-like behavior, but in a benign form."
Eckelberry took exception with using Spycar in general, and with using only Spycar specifically.
"It's not a serious testing tool," he claimed. "The mantra of any type of security test is that you have to test against real-world scenario. Relevancy is critical."
Spycar fails on those points, he went on. "It does things like install fake registry keys, changes your start page and the like. It is specifically designed to test how well anti-spyware programs block unknown applications, not [how they] scan and remove."
Randy Abrams, for 7 years the man responsible at Microsoft for ensuring that all software it released was malware free, was even more blunt. "I was livid about the testing [Consumer Reports] did. They tested anti-spyware software without ever testing how it detected and removed spyware."
F-Secure's Anti-Spyware and Webroot's Spy Sweeper 4.5 (now superseded by 5.0) tied for first in Based Consumer Reports' tests with matching scores of 89 out of a possible 100. Sunbelt's CounterSpy clocked in at seventh with a score of 70, while Microsoft's free Windows Defender came in dead last with a score of 43.
Eckelberry said it wasn't sour grapes over a low score that lead him to take on Consumer Reports' testing.
"No test is perfect," he said. "But there are certainly degrees [of imperfection]. It should be all about relevancy, but here it's not."
Consumer Reports' September issue was already under fire over its anti-virus testing procedures when Eckelberry raised the flag on anti-spyware. Within days of the issue making the newsstand, McAfee posted an open letter taking the publication to task for hiring a lab to create 5,500 new variants derived from half a dozen malicious code categories.
"Creating new viruses for the purpose of testing and education is generally not considered a good idea," wrote McAfee's Igor Muttik in an entry on the security company's Avert Labs blog. "Viruses can leak and cause real trouble."