Rolling Review: WatchFire Blazes Past Competition

AppScan was the only product in this Rolling Review to handle Ajax, and it did it without the gotchas that plagued rivals. Now that's hot.
Not only is AppScan the most mature Web application vulnerability scanner on the market, developed in 2000 as a companion to Sanctum's AppShield Web application firewall, but now IBM owns it as a result of its Watchfire acquisition.

We weren't sure AppScan's experience would be enough: The Ajax apps we've been feeding our scanners have proved troublesome, even for long-established products. However, AppScan impressed us with its ease of use and advanced functionality and reliability. It was the most successful so far at traversing our Ajax apps.

There were some tense moments though. When we scanned a sample Ajax app, AppScan was unable to automatically parse the JavaScript and enumerate the entire application. When Watchfire investigated, however, it said it had no trouble with the application. What gives?

CLAIM:  Web application scanners in this Rolling Review must not only find traditional vulnerabilities, like XSS and SQL injection flaws, but also handle Ajax applications, in which part of the app is running locally in the browser.

CONTEXT:  Complex Ajax apps represent a new and, we found out, highly challenging twist for these products, but we don't recommend purchasing a scanner that isn't able to handle Web 2.0 environments, given that so much future development is moving in that direction. And Web application scanners should be just one element in a comprehensive, layered program--educating developers and integrating security reviews into the development life cycle are just as crucial.

CREDIBILITY:  AppScan sets the standard for features, usability, and reliability. While not perfect, it's the pacesetter and the first product evaluated to successfully traverse our Ajax apps.
The problem was traced to a rogue Microsoft XML library on our scanner that wasn't properly registering. Until the exact cause was tracked down, Watchfire added a temporary fix to the AppScan installer to ensure that the library worked properly. Customers needing JavaScript testing functionality should be sure they're running the AppScan Update tool to get the fix for this bug.

Once that issue was resolved, AppScan performed as promised, becoming the first Web application scanner we tested to properly identify Ajax functionality and navigate those sections of the app other scanners failed to automatically crawl.

AppScan encapsulated the best features of all the products in this review with few of the faults. It's cleanly designed, and the interface was easy to use, much as with Hewlett-Packard's WebInspect, but without its reliability questions.

For advanced users, AppScan's built-in utilities are nearly on par with the rich suite of tools integrated into WebInspect. Additionally, since version 7.5, AppScan has taken a cue from the Firefox browser, letting users develop extensions that can integrate into the product. These add-ons reflect the growing popularity of open source products and communities. One sample extension is a complete development environment in itself, integrating the popular open source scripting language Python with AppScan's core engine.

Much of the value in a scanner stems from how seamlessly it can be incorporated into your workflow to provide meaningful and actionable data throughout the development process. Exposing the product via extensions is a great way to let customers use AppScan in a way that best fits their particular needs or environment. Take the Pyscan module. An organization might implement custom scans of different branches of an application under development by automatically scripting both scanning and reporting as code is forked for reuse or checked into a source-code repository. Having the simplicity and popularity of Python tied to the scanning engine that makes AppScan tick is a powerful combination. Creative types will discover a variety of potential uses.

All this isn't to say that AppScan is flawless. Scanning an Ajax Web mail app was problematic until someone in the AppScan support group figured out that the Ajax application was detecting the User-Agent changing between the initial recorded log-in session and the rest of the requests, dumping the scanner back to the log-in screen. Once the cause was identified, it was a simple tweak to change the scanner's User-Agent to be consistent with the login script.

We were pleased by Watchfire's responsiveness. We have no reason to think the company won't be just as attentive to enterprise IT groups. Bottom line: AppScan isn't perfect, but it's more reliable, functional, and extensible and easier to use than any other product we've tested so far.

IBM's Watchfire AppScan

$24,000 for a perpetual license. Maintenance is $4,800 per year, first year not included.

We're testing Ajax-capable application scanners at our Real-World Labs at the University of Florida. We're assessing reliability, advanced features, ease of use for nonsecurity personnel, ability to map and scan Ajax functionality, prevalence of false positives and ease of manual adjustments and product updates to address them, prevalence of false negatives, and price. We'll evaluate SaaS offerings, too, though not on ease of use and advanced features.

Acunetix Web Vulnerability Scanner

Hewlett-Packard WebInspect (formerly from SPI Dynamics), Cenzic Hailstorm, N-Stalker Web Application Security Scanner

ASyhunt Technology and WhiteHat Security. Contact the author at [email protected] for consideration.

InformationWeek Labs Rolling Reviews present a comprehensive look at a hot technology category, beginning with market analysis and wrapping up with a synopsis of our findings.
Click here to see our kickoff to this Ajax-capable application scanner series.

Editor's Choice
Mary E. Shacklett, President of Transworld Data
James M. Connolly, Contributing Editor and Writer