Sophos has patched seven vulnerabilities in its antivirus software, including bugs that could be used by an attacker to take control of a Windows, Mac, or Linux system.

By exploiting the vulnerabilities, an attacker may be able to gain control of the system, escalate privileges, or cause a denial-of-service condition, according to a related security bulletin released the U.S. Computer Emergency Readiness Team (US-CERT).

The vulnerabilities were identified by Tavis Ormandy, a security researcher at Google, after he reverse-engineered the Sophos antivirus application in his spare time. "By design, antivirus products introduce a vast attack surface to a hostile environment. The vendors of these products have a responsibility to uphold the highest secure development standards possible to minimize the potential for harm caused by their software," said Ormandy in a related research paper, "Sophail: Applied attacks against Sophos Antivirus."

[ Tempted to strike back by hacking a hacker? Read this first: 9 Facts: Play Offense Against Security Breaches. ]

Ormandy said the paper focuses on "the process a sophisticated attacker would take when targeting Sophos users," noting that it applies to all platforms that Sophos supports, including Windows, Mac, Linux and their SAVI SDK product. SAVI SDK refers to the software development toolkit that Sophos OEM partners can use to integrate its antivirus application into other security software.

Graham Cluley, a senior technology consultant at Sophos, Monday confirmed the vulnerabilities, and said Sophos has seen no in-the-wild attacks that exploit the bugs. In a blog post, Cluley also commended Ormandy's "responsible approach" to bug disclosure, noting that Sophos was informed of the vulnerabilities prior to the researcher detailing them publicly, which gave it time to patch most of them.

All told, Ormandy identified eight previously undocumented vulnerabilities. The first was reported to Sophos on September 10, 2012, and the most recent on October 5. Sophos said it began releasing fixes for the issues in October, and by Monday had issued patches for all but one of the vulnerabilities.

The two most critical bugs -- both now patched by Sophos -- stemmed from the manner in which the Sophos AV engine scans files that were compiled using Visual Basic 6, as well as malformed PDF files. Both bugs could be exploited by attackers to run arbitrary code on targeted PCs.

Other vulnerabilities patched by Sophos include a Web protection and blocking page that included a cross-site scripting flaw, a bug relating to how the Sophos AV buffer overflow protection system interacts with address space layout randomization (ASLR) -- present in all versions of Windows starting with Vista -- and errors relating to how Sophos AV handles CAB and malformed RAR files, either of which could lead to memory corruption errors.

The sole unpatched vulnerability discovered by Ormandy relates to a scanning problem. "Tavis Ormandy has provided examples of other malformed files which can cause the Sophos anti-virus engine to halt -- these are being examined by Sophos experts," said Cluley, who reported that the company had seen no evidence of this occurring in the wild. Interestingly, Apple users of the free Sophos AV product have reported that scans can regularly cause their Macs to hang, seemingly after encountering malformed files.

Ormandy has made a hobby out of investigating the Sophos antivirus software. Last year, he reverse-engineered the core AV engine in Sophos Antivirus 9.5 for Windows. At the time, Ormandy criticized the Sophos software for employing poor buffer-overflow protection and cryptography, and for including a host-intrusion prevention system that was compatible only with Windows XP and earlier versions of Windows.

From a coding standpoint, how does Sophos antivirus software compare with the competition? That question is difficult to answer, since Ormandy studied only one antivirus vendor's product, but with luck, his research will inspire others to undertake similar investigations of other antivirus products.

As for Sophos, however, Ormandy's research raises troubling questions. For example, why does a firm that sells security software seem to have side-stepped secure coding practices and failed to embrace modern attack-mitigation technologies, such as ASLR?

Many of the discovered vulnerabilities "could have been severely limited by correct security design, employing modern isolation and exploit mitigation techniques," said Ormandy. "However, Sophos either disables or opts out of most major mitigation technologies, even disabling them for other software on the host system. This makes the exploitation process straightforward, providing a homogeneous exploitation environment conducive to wide-scale attack."

According to Ormandy, after he notified Sophos of the bugs he'd discovered, the company requested that he withhold publishing the details until it had time to release related patches, and he agreed to do so. "Sophos [was] able to convince me they were working with good intentions, but they were clearly ill-equipped to handle the output of one cooperative security researcher working in his spare time," he said. "They told me they will work on this and will improve their internal security practices." No doubt a third research report from Ormandy in a year's time will review the company's results.

Attackers are increasingly using a simple method for finding flaws in websites and applications: They Google them. Using Google code search, hackers can identify crucial vulnerabilities in application code strings, providing the entry point they need to break through application security. In our report, Using Google To Find Vulnerabilities In Your IT Environment, we outline methods for using search engines such as Google and Bing to identify vulnerabilities in your applications, systems and services--and to fix them before they can be exploited. (Free registration required.)

Federal agencies must eliminate 800 data centers over the next five years. Find how they plan to do it in the new all-digital issue of InformationWeek Government. Download it now (registration required).