DARPA Pursues Deep Web Search Tools

6 Cool Apps From Uncle Sam

The Defense Advanced Research Projects Agency (DARPA) wants to develop the next generation of search technologies, particularly those that can help the military and government agencies find and organize publicly available information on the Internet.

To address the challenges of a one-size-fits-all approach used by today's search engines, DARPA has kicked off a program called Memex, short for memory and index. The program aims to "revolutionize the discovery, organization, and presentation of search results" and "extend the reach of current search capabilities," according to DARPA. Memex will focus on three technical areas: domain-specific indexing, domain-specific search, and applications that pertain to the Department of Defense (DOD).

"We're envisioning a new paradigm for search that would tailor indexed content, search results, and interface tools to individual users and specific subject areas, and not the other way around," DARPA program manager Chris White said in a written statement.

[Memex is just one of DARPA's data development projects. Read DARPA Opens Software, Data To Public.]

DARPA believes current search capabilities miss information in the "deep Web" -- the parts of the Internet that aren't indexed by commercial search engines. They also overlook shared content across Web pages. The agency envisions using commodity hardware and an open-source architecture to build this advanced search technology that would be capable of cross-referencing information more quickly and efficiently.

Memex was inspired by a hypothetical device described in "As We May Think," a 1945 article for The Atlantic Monthly written by Vannevar Bush, a World War II-era director of the Office of Scientific Research and Development (OSRD). DARPA said it wants to improve the ideas described in that article, with the creation of domain-specific Web content indexing and search.

Initially, the program will focus on fighting human trafficking, a problem that touches various parts of government, including the military, law enforcement, and intelligence agencies. The commercial sex trade in particular has garnered significant Web presence via forums, chats, advertisements, and job postings. "An index curated for the counter-trafficking domain, along with configurable interfaces for search and analysis, would enable new opportunities to uncover and defeat trafficking enterprises," DARPA said.

Earlier this month, DARPA issued a solicitation for research on the subject. Research proposals, due on April 8, must address approaches that "enable revolutionary advances in science, devices, or systems," the agency said. Procurement contracts and cooperative agreements, but not grants, will be awarded to the winners. DARPA will hold a conference on February 18 to discuss the technical details of Memex with program participants.

Solid state alone can't solve your volume and performance problem. Think scale-out, virtualization, and cloud. Find out more about the 2014 State of Enterprise Storage Survey results in the new issue of InformationWeek Tech Digest.