DARPA Pursues Deep Web Search Tools - InformationWeek
Government // Leadership
09:30 AM
How Upwork Cut Zero-Day File Attacks by 70%
Oct 05, 2017
Upwork has millions of clients and freelancers that have to upload and download many files to and ...Read More>>

DARPA Pursues Deep Web Search Tools

Defense Advanced Research Projects Agency project Memex will shine light on parts of the Web where commercial engines don't search.

6 Cool Apps From Uncle Sam
6 Cool Apps From Uncle Sam
(Click image for larger view and slideshow.)

The Defense Advanced Research Projects Agency (DARPA) wants to develop the next generation of search technologies, particularly those that can help the military and government agencies find and organize publicly available information on the Internet.

To address the challenges of a one-size-fits-all approach used by today's search engines, DARPA has kicked off a program called Memex, short for memory and index. The program aims to "revolutionize the discovery, organization, and presentation of search results" and "extend the reach of current search capabilities," according to DARPA. Memex will focus on three technical areas: domain-specific indexing, domain-specific search, and applications that pertain to the Department of Defense (DOD).

"We're envisioning a new paradigm for search that would tailor indexed content, search results, and interface tools to individual users and specific subject areas, and not the other way around," DARPA program manager Chris White said in a written statement.

[Memex is just one of DARPA's data development projects. Read DARPA Opens Software, Data To Public.]

DARPA believes current search capabilities miss information in the "deep Web" -- the parts of the Internet that aren't indexed by commercial search engines. They also overlook shared content across Web pages. The agency envisions using commodity hardware and an open-source architecture to build this advanced search technology that would be capable of cross-referencing information more quickly and efficiently.

Memex was inspired by a hypothetical device described in "As We May Think," a 1945 article for The Atlantic Monthly written by Vannevar Bush, a World War II-era director of the Office of Scientific Research and Development (OSRD). DARPA said it wants to improve the ideas described in that article, with the creation of domain-specific Web content indexing and search.

(Image: Montage courtesy of DARPA)
(Image: Montage courtesy of DARPA)

Initially, the program will focus on fighting human trafficking, a problem that touches various parts of government, including the military, law enforcement, and intelligence agencies. The commercial sex trade in particular has garnered significant Web presence via forums, chats, advertisements, and job postings. "An index curated for the counter-trafficking domain, along with configurable interfaces for search and analysis, would enable new opportunities to uncover and defeat trafficking enterprises," DARPA said.

Earlier this month, DARPA issued a solicitation for research on the subject. Research proposals, due on April 8, must address approaches that "enable revolutionary advances in science, devices, or systems," the agency said. Procurement contracts and cooperative agreements, but not grants, will be awarded to the winners. DARPA will hold a conference on February 18 to discuss the technical details of Memex with program participants.

Solid state alone can't solve your volume and performance problem. Think scale-out, virtualization, and cloud. Find out more about the 2014 State of Enterprise Storage Survey results in the new issue of InformationWeek Tech Digest.

Elena Malykhina began her career at The Wall Street Journal, and her writing has appeared in various news media outlets, including Scientific American, Newsday, and the Associated Press. For several years, she was the online editor at Brandweek and later Adweek, where she ... View Full Bio

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Author
2/12/2014 | 10:32:10 PM
Catching up
Given DARPA's annual budget is dwarded by Google's and Microsoft's resources, it should be interesting to see what comes of DARPA's efforts to develop a deeper search tool.

Thomas Claburn
Thomas Claburn,
User Rank: Author
2/12/2014 | 7:14:55 PM
the dark web
If DARPA is missing too much of the deep web, someone could always ask the NSA...
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of IT Report
In today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll