Anti-Spyware Vendors Mad About Consumer Reports Test Methods

Vendors including Microsoft and Sunbelt Software say the consumer magazine's test is bogus because it doesn't take into account how security software detects and removes threats.
Consumer Reports defended its actions by saying it was the best way to test anti-virus software against "novel threats not identified on signature lists."

McAfee and others, including Abrams, now the director of technical education at Eset, a Slovakian security company, rejected that claim. Instead, they pointed out that "retrospective" testing -- where an anti-virus product is installed but then not allowed to update its definitions for several weeks or months -- is a more realistic way to test how software handles new threats. Products can be objectively ranked, said Abrams, when new-found malware is tossed at out-of-date definition files. The practice quickly shows how well each anti-virus program sniffs out new attacks using its behavior-based tools.

"I thought that their anti-virus testing was bogus and useless," said Abrams. "But their anti-spyware testing was worse."

Consumer Reports did not respond to a call for comment, but in a letter from Dean Gallea, its test program leader, to Eckelberry that that latter shared with TechWeb, the publication didn't sound like it was about to change its mind, or its anti-virus and anti-spyware conclusions.

"Thanks for your insights on the use of behavior simulation to test the performance of anti-spyware programs. We believe we understand your concerns, however we chose this approach because we felt it best captured the flexibility of the software," Gallea wrote.

"We are constantly re-evaluating our test program, and will take these and other considerations into account in future tests," he added.

"I grabbed a copy [of Consumer Reports]," said Abrams. "On the cover it said 'sleeping pills, the facts they don't tell you,' or something along those lines. But if they do such an incredibly bad job in testing consumer anti-virus and anti-spyware, how could I ever trust them with something medical related? I was completely dumbfounded by the whole thing," he added.

"I am happy that NOD32 [Eset's anti-virus title] was not tested. There is no honor in claiming that you were number one in a worthless test. It would be like being rated best food by a critic who has no taste buds and doesn't understand nutritional content."

Editor's Choice
Brian T. Horowitz, Contributing Reporter
Samuel Greengard, Contributing Reporter
Nathan Eddy, Freelance Writer
Brandon Taylor, Digital Editorial Program Manager
Jessica Davis, Senior Editor
Cynthia Harvey, Freelance Journalist, InformationWeek
Sara Peters, Editor-in-Chief, InformationWeek / Network Computing