Hackers are adding virtual machine detection to their worms and Trojans to stymie analysis by anti-virus labs, a security research said Sunday.

The tactic is designed to thwart researchers who use virtualization software, notably that made by VMware, to quickly and safely test the impact of malicious code. Researchers will often run malware in a virtual machine to protect the system's actual operating system from infection; virtualization software also lets analysts test malware against multiple operating systems on a single computer.

"Three out of 12 malware specimens recently captured in our honeypot refused to run in VMware," said Lenny Zeltser, an analyst at SANS Institute's Internet Storm Center (ISC) in an online note Sunday.

Malware writers use a variety of techniques to detect virtualization, including sniffing out the presence of VMware-specific processes and hardware characteristics, said Zeltser. "More reliable techniques rely on assembly-level code that behaves differently on a virtual machine than on a physical host," he added.

Researchers can fight back, Zeltser said, by patching the malicious code so that the virtual machine routine(s) never executes, or by modifying the virtual machine to make it more difficult for malware to detect that it's running in a virtual environment.

Two other ISC researchers, Tom Liston and Ed Skoudis, spelled out anti-detection techniques at a recent SANS conference. The paper can be downloaded from the ISC site as a PDF file.