More details about one of the world's most ambitious--and controversial--database projects will be released this week when the Defense Advanced Research Projects Agency issues a report to Congress on its Total Information Awareness program. The initiative has been a point of contention for privacy groups and others worried that Darpa or other government agencies will collect personal information on U.S. citizens in a national security effort to thwart terrorists through sophisticated data-analysis and collaboration techniques. If Darpa can overcome the privacy concerns, some experts say, its bleeding-edge prototype system just might work. "We're at the cusp of actually being able to do it," says data-warehousing consultant Bill Inmon, citing advances in storage, database, and microprocessor technologies, accompanied by continuing declines in hardware costs, that make such a large-scale undertaking conceivable.

Total Information Awareness represents a data-management challenge as bold as it is controversial. The goal, as Darpa's Information Awareness Office (headed by John Poindexter, former national security adviser in the Reagan administration) envisions it, is to sift through vast quantities of data contained in government and business databases, detect suspicious patterns of activity, identify the shady characters behind those actions, then find them before they can do any harm. The five-year research and development project would test the limits of database integration and scalability and require breakthroughs in language translation, pattern matching, and agency-to-agency collaboration. "It is anticipated this will require revolutionary new technology," Darpa said last year in a document soliciting bids from potential suppliers.

How would it work? According to public comments made by Darpa officials and information on the agency's Web site, the goal is to create an architecture capable of building a huge database that gets populated, with the help of automated processes, from existing databases. A database crawler would be used to recognize the data structures of source databases, facilitating the flow of data. Data-mining algorithms would be run against the data collected and models generated in an attempt to predict terrorist behavior. Collaboration tools would link experts together for quick action when a terrorist's "signature" is suspected.

"One goal is to develop ways of treating the worldwide, distributed legacy databases as if they were one centralized database," Poindexter said last year. Total Information Awareness was launched in fiscal 2003 with $10 million in initial funding, but related projects predate the terrorist attacks of Sept. 11, 2001. Darpa will spend an estimated $240 million on the combined projects from 2001 through 2003. Prototypes of the system would be turned over to other Department of Defense agencies for adoption.

Illustration by Michael Morgenstern