Defense Intelligence Agency Boosts Search Firepower

The U.S. Defense Intelligence Agency will within the next month expand its use of an emerging search technology that improves the ability of the military, defense policy makers, and combat strategists to make more informed decisions, officials said Thursday.

It's the latest step in the agency's efforts to move beyond basic keyword search engine capabilities and apply search technologies that better help its personnel connect the dots.

Analysts and information gathers working for the DIA, a Defense Department combat support agency and a major producer and manager of foreign military intelligence, today can use Endeca Technologies' Information Access Platform to search 20 different sources of intelligence gathered by agents in the field.

Weeks from now the platform's reach will be extended to include intelligence gathered through the interception of radio and other signals and news feeds such as Reuters as well as message traffic from the State Department, "our largest authoritative repository of database information about foreign military capabilities," said Lewis Shepherd, chief of requirements and research for the DIA, which has 11,000 military and civilian employees worldwide.

When searching for information, DIA personnel may have as many as 300 individual data feeds, databases, and data from other intelligence agencies they can access. The 20 sources available today through Endeca represent more than half of all the agency's data, and the agency plans to add more over the next two years. Other sources can be searched using search technology from Autonomy Corp., Google, and Vivisimo Inc.

Shepherd and his staff first met with Endeca in 2004 because the agency needed a more advanced way of extracting useful information from an enormous volume of data. Prior to using Endeca, responding to a specific query from Congress or another agency could take as long as five days. Now it takes minutes. At the time, Endeca's guided navigation approach was unique in terms of the methods it used for finding information, Shepherd said.

Today, there's no shortage of search technologies out there. What continues to appeal to the DIA is Endeca's ability to offer users guided navigation as part of the search summaries it delivers. Endeca uses what the company calls a "faceted classification system" to read meta data tags on all of the data flowing through its systems and sort query results by any category in any order. The agency created these data tags using a number of different XML formatting and meta data tagging software packages, including those from Attensity and Inxight Software as well as Lockheed Martin's AeroText and SRA International's NetOwl.

Endeca's searches come back to the user in a series of menus. "It's almost like having a librarian organize your data and summarize what's on your shelves for you," said Endeca CEO Steve Papa. If one was to search on the late singer Frank Sinatra, for example, the search results might be broken down into a number of categories, including year, record labels, and movies, as opposed to a Google-like listing of all keyword matches. (In Sinatra's case, that's more than 2.7 million matches).

More specific to the DIA, when an analyst searches for information related to a "tank," the search engine is able to understand that, if the analyst is also searching for information about artillery, then "tank" refers to a vehicle rather than a container of water. The search results will be prioritized to reflect this. The DIA first went live with its Endeca-based search system in early 2006 by providing its intelligence gatherers and analysts with access to any intelligence gathered by field staff in the form of reports. Thanks to Endeca's ability to place search findings into categories, the DIA not only improved its search capabilities, "we also see we're missing entire areas of information that we need to collect," Shepherd said.

"Search injects a note of serendipity into the information finding business," said Sue Feldman, IDC's VP for search and digital marketplace technologies. "Exact matching, like in a database application, only gets you what you ask for. In the defense setting, what you're looking for is what you don't know enough to ask for."

Workers spend up to 10 hours each week trying to find information, and they waste half of that time by not being able to find it, Feldman said, adding, "In the case of the DIA, this could have serious consequences."

The use of advanced search technologies will continue to grow in the intelligence community. The DIA is now using its experience with Endeca to work with the Office of the Director of National Intelligence to improve the navigability of that organization's defense intelligence. Endeca is part of the DIA's larger services oriented architecture approach called ALIEN, or all-source intelligence environment, started last summer.

The lessons learned from intelligence failures of the past are hard to ignore. "We take very seriously the mandate of the 9/11 Commission and the WMD commission to do vastly better intelligence gathering and analysis," Shepherd said.

With so many U.S. troops in the field counting on the DIA, the agency couldn't have picked a better time to expand its search capabilities.