Testing's New Wave

Binary code instrumentation automates testing of Windows apps

By Rich Levin

oftware testing for Windows applications is about to become more interesting, productive, and fun thanks to a relatively new and largely unknown Windows executable file format from Microsoft, and the breakthrough research of a Digital Equipment spin-off called TracePoint Technology Inc.

Small numbers of application developers are enjoying what could be the first wave of fully automated testing tools. Although many software testing tools are characterized by their purveyors as automated, in truth, nearly all require extensive preprocess preparation and programming by at least one member of the IT development team.

Regression tests, for example, require scripting, w hich means programmers or quality-assurance developers literally build separate programs, called test scripts, that are run against the program being tested. The problem is that even the test scripts contain bugs and must be tested. In addition, many scripts fail to exercise all of a program's interface elements.

Similar issues exist for users of code coverage and performance profilers. These tools require programmers to manually instrument their application source code. Developers or quality-assurance personnel must manually pepper the program code with coverage or profiling directives, known as probes, which are then interpreted or otherwise acted upon by the testing tools at run time.

In all cases, while software quality and run-time performance nearly always improve when testing tools are integrated into the development and quality-assurance processes, programmer productivity plummets as a result of the additional care and feeding required by the testing tools.

"You have to actually modify you r source code, which can kill your productivity in three different ways," says Maribeth Ahne, an associate programmer with printer manufacturer Lexmark International Inc. in Lexington, Ky. "I recently instrumented my code and couldn't get it to build. Once I got it to compile, some critical timings were off, because I had modified the source code."

Love-Hate Relationship
Given the choice between learning and adhering to a test methodology and watching paint dry, developers are more likely to choose the latter. The truth is, most developers love to hate software testing. That's why most development teams forgo testing, relying instead on their own debugging and intuition to optimize their new applications.

"The heavy, methodology-centric kind of stuff has a very limited appeal," says Dick Heiman, research manager for application development tools with International Data Corp., a technology market research firm in Framingham, Mass. There's a place for methodology, he says. It prescribes what has to happen as part of the development process. But in general, the software world doesn't work that way, he adds. "If you're a nuts-and-bolts programmer, working with RAD tools and tight, get-it-done-now deadlines, then you need high-productivity testing tools," Heiman says.

In at least two key testing categories-code coverage and code profiling-those high-productivity testing tools have arrived. A new technology has emerged that marries the benefits of testing with the ease of complete automation. Automatic binary code instrumentation scans and robotically instruments 32-bit Windows executables, without requiring any programmer intervention.

By targeting native Windows executables, as opposed to source code or intermediate files, these new products are independent of language, source code, compiler, and hardware. Provided the development tool spits out 32-bit Windows Portable Executable (PE) files, its applications can be automatically instrumented and tested.

The first products to incorporat e binary code instrumentation were developed by TracePoint, a San Jose, Calif., startup launched in February by Digital. The company's key technology personnel were gleaned from Digital's Western Research Laboratories, which at the time was working on a similar technology for use with Unix COFF (Common Object File Format) files.

While COFF, originally developed five years ago at Digital's Western Research facilities for Digital Alpha Unix, was supposed to become the standard file format for Unix systems, vendors failed to strictly adhere to the specification. This fragmentation prevented developers from building tools that could exploit COFF on a universal, cross-platform basis.

When they realized that COFF was unworkable, researchers turned their attention to the Windows platform.

The new target became an as-yet unexploited, fully standardized file format: Microsoft's Portable Executable. Introduced with Windows NT and extended to Windows 95, PE is the standard binary file format for all Win32 ex ecutables and shared libraries on the Intel platform.

Before PE, Win16 applications for Windows 3 used the NE (New Executable) format, which was primitive and not too different from MS-DOS's original MZ format, says Amitabh Srivastava, TracePoint's chief technology officer and engineering VP, who is the programmer largely responsible for inventing the company's automatic binary code instrumentation. "With PE, you know what the format looks like, what it contains, and where to find things. You don't have to care what language or compiler was used to create the file."

TracePoint's two new testing tools-HiProf, a performance profiler, and Visual Coverage, a test-coverage monitoring and reporting utility-are currently limited to Windows compilers that generate pure, standalone PE executables and shared dynamic link libraries. Languages such as Microsoft's Visual Basic, Powersoft's PowerBuilder, Sun Microsystems' Java, and in some cases, Borland's Delphi, which require ancillary run-time libraries or vir tual machines, are not supported.

No Coding
Still, early adopters of TracePoint's tools for Windows C++ development say the technology is a godsend. "Basically, you point it at the executable, and then it starts chugging," says Dave Wilson, an application development specialist with Frank Russell Co., a Tacoma, Wash., financial services firm managing over $100 billion in assets. "There's no coding whatsoever. You just watch the screen. It takes a little while, but when it's done, it has instrumented the executable."

Depending on the tool being used, developers can perform different tests after the executables are instrumented. With HiProf, instrumented applications typically undergo regression testing, or they are put through their paces by a programmer or quality-assurance engineer. HiProf monitors nearly every aspect of the application's performance, and generates reports that isolate performance bottlenecks.

"I'm always surprised at the bottlenecks I find," Wilson says. "We recently profiled an application, and the most time was spent on the calls to INI configuration files. So we took some out, and achieved a 30% performance increase. You never know until you profile."

HiProf goes further than manual profilers in other ways, too: Complete application inspection, and detailed reporting. Where most profilers time applications at the routine or the function level, HiProf uses hierarchical analysis, a practice required to deliver incontestable runtime data on object-oriented applications, such as those crafted in C++.

Not only does HiProf reveal the amount of time that a method took, it also reveals the amount of time that the operating system took, and the amount of time third-party components and libraries took, Wilson says. "You get information on the parent processes; you get information on the child. When you instantiate an object, it's hard for a profiler to understand what's going on, and this one does."

The other tool in TracePoint's lineup is Visual Coverage, an automa ted code-coverage system. Code coverage validates that an entire application has, in fact, been tested. Unless they apply code-coverage technology to an application-development project, team leaders can never be sure their regression and other tests have actually thoroughly tested the project.

"The issue of testing is that you only get the answers to the questions that you ask," notes IDC research manager Heiman.

Code coverage lets you know how much of the software your tests have executed, says Lexmark's Ahne. "It determines how much your test is covering of your executable, not exactly to determine how robust your executable is. There are products that tell you how robust your code is, locate memory leaks and pointer problems, and so on-but that's not what I'm trying to do. I need to know how much of the program we are actually testing, because we want to test all of it."

Ahne, a testing specialist, says she finds it surprisingly easy to miss key application components in testing when coverage t ools are not used. She says she battled for months with conventional utilities, manually instrumenting her teams' code, with little to show for it. "I struggled for two months, and still didn't have accurate results," she says. "Four days after downloading Visual Coverage from the Web, I was able to get accurate code-coverage results."

Ahne credits the product's absolute automation of the coverage process for her success. All a developer must do is use a debug build, which adds the required symbolic information to the PE files. Then the developer points Visual Coverage to that EXE or DLL, modifies a few settings if the defaults aren't suitable, and chooses Instrument, then starts the process. Visual Coverage instruments the files, and the developer just runs the application, or test scripts, with it.

Dead-Code Finder
There is also an unexpected benefit of automated code coverage: locating dead or unreachable code. When writing in C++, when a programmer develops a class, he may not use ever y method, and may create classes that are ultimately never used. With a 250,000-line piece of software, it's nice to have a report that says, "This stuff is dead; this stuff is unreachable."

For the time being, the downside to TracePoint's products is their inability to deal with output from languages other than C++. Company officials say they'll address that soon.

TracePoint plans to announce a Visual Basic product later this year that will automatically instrument, profile, and cover Visual Basic binaries, and will be able to handle hybrid VB applications, which consist of a combination of VB and C++ code.

Sources say a Java version is slated to follow the VB support. If TracePoint's automatic binary code instrumentation resonates with Windows developers, it's a safe bet that other testing tool vendors will follow. Testing suites that focus more on productivity and less on methodology, may not be far behind.

Back to News in Review

Send Us Your Feedback

Top of the Page