Keep Attackers At Bay

The problem with behavioral monitoring is that applications aren't static. New versions are rolled out, and patches and software fixes are continuously deployed. That means these behavioral monitors must be retrained for application changes or new application deployments, a process that may require significant effort.

"In order for the Sana agent to learn, it had to see certain functionality in regard to user and application behavior," Network Health's Berry says. "The internal Web testing team had to exercise all the functions of the application to know what was good and what was bad." After deploying patches on servers, Berry switches the agents from protect mode to learning mode as part of a rigid change-control process.

Buffer overflows are a common software vulnerability exploited by malware. A buffer is a segment of memory used to hold data while the data is being processed. Attackers often use buffer overflows as the first step in executing their own code on a target computer.

Cisco, McAfee, and Sana's behavioral-analysis agents count buffer-overflow protection as one of their core offerings and have been the standard protection mechanism for years. Once the malicious code included with a buffer overflow begins to execute, these agents will detect unusual system calls and prevent the code from running, thus preventing an attacker from gaining control of the target.

But these products don't prevent the buffer from being overflowed with too much data. They wait until the malicious code begins executing before acting. If the buffer overflow itself causes the application or computer to crash, the behavioral engine won't be able to stop the crash.

A new technology addresses this problem by dealing with the buffer overflow at a more targeted level inside the operating system. The Memory Firewall from startup Determina Inc. grew out of research conducted at MIT. The research demonstrated that buffer overflows violate basic programming conventions that are common among all compiled applications. Rather than analyze system calls for potential buffer-overflow activity, Memory Firewall enforces these conventions.

Specifically, when a program executes, Memory Firewall loads all possible instructions for that program into memory. At this point, the product creates a small virtual-machine environment to control the program's execution. Because all the instructions of the program have been loaded into the virtual environment, Memory Firewall can verify which instructions should come next and that the instructions are part of the original program loaded at startup.

"The issue with system call monitors is that they aren't watching every single instruction in the user space," says Charles Renert, head of security research and development at Determina. Attacks aren't blocked until the malicious code executes and then issues the first system call. By contrast, the Memory Firewall doesn't allow a single instruction of malicious code to run. In addition, while the Memory Firewall doesn't prevent the buffer from overflowing, in most cases it will keep the application or computer from crashing.

Once a block of code is inspected, it's moved to a secure cache so that it doesn't have to be reinspected. The software will add up to 15% latency to an application in the first few seconds of operation, Renert says. Out of the box, the Memory Firewall protects all Windows services and server applications. Administrators also can add custom applications.

At Sappi Fine Paper, Cupps used the Memory Firewall for buffer-overflow protection and Entercept for protection against SQL injection and directory traversal attacks on the same IIS Web servers. He also extensively tested Cisco Security Agent. The buffer-overflow feature was turned off in Entercept.

One reason Cupps did this was that he preferred Determina's approach to buffer-overflow protection. But just as important was that Memory Firewall needed little configuration. "We get a high level of protection [from the Memory Firewall] with almost no oversight and a reasonable cost," Cupps says.

"With CSA, you have to choose what you want to allow to happen. With Entercept, you run it in learning mode, choose things not to alarm for, and establish a baseline. And whenever you change the baseline, you have to touch Entercept again," he says. "With Memory Firewall, you don't worry about that."

Still, Memory Firewall protects only against memory-based attacks such as buffer overflows, Cupps says. It doesn't provide the complete range of protection against malware and other attacks offered by Entercept, Primary Response, and Cisco Security Agent, which is why Cupps paired Memory Firewall with Entercept. Lesson learned: When it comes to protecting your network, it's best to cover all bases.

Illustration courtesy of Andrew Shachat/Veer