In Search of Intelligent Search

Is it possible to have shorter hit lists and fewer tools yet better intelligence? Yes. Here's how organizations are helping users tap into more sources without forcing them to waste time dealing with redundant search engines.

To address the proliferation problem, organizations typically take one of two approaches: consolidate disparate search tools by choosing a single enterprise system, or aggregate the results of the various search engines within a federated search system (many so-called enterprise search systems include federation in their suites).

Consolidating on a single, enterprisewide search tool presents the possible drawback of forcing users to give up the tools they're familiar with, but few if any enterprises eliminate old tools in favor of single, more powerful search engines (see "Opinion: Enterprise Search Is a Myth" below). Without proper training and support promoting user adoption, the enterprise tool can end up just adding to the long list of enterprise search engines.

Federation, which is a capability available in several leading enterprise search products, can present accuracy and redundancy problems, according to Reynolds. If each individual engine contributing to the aggregated search turns up its share of inaccurate results, "you might be breaking down silos, but you're creating a new kind of info-glut," he points out. "If you have 10 times the amount of inaccurate information, what have you gained by using federated search?"

Mentor Graphics chose the enterprisewide search route, and it gained broad user acceptance by targeting a pain point. The design automation software company used to rely on separate search functions on the intranet and the consumer Web site depending upon whether staff or customers wanted to search the knowledge base, online documentation or internal document repositories. These siloed search tools made it tough to find answers to technical questions. When Mentor purchased a competitor and needed to upgrade its customer support infrastructure in early 2004, it looked for a single search solution that would support internal needs and Web customer self-service.

After narrowing the field to four vendors, Mentor tested each product on its own content by posing a set of 1,500 questions and evaluating the answers. After reviewing the results, the company chose InQuira to power its "SupportNet" online system. InQuira's software handles both content and structured data, and it includes a semantic processing engine that is said to understand and respond to intent-even the user is struggling to find the proper search terms.

Internal users were frustrated with the separate search silos, so they readily accepted the new system. Customer response also has been positive, says Mentor VP of worldwide customer support Tom Floodeen, and he points to increased Web traffic and declining call center loads as evidence of that acceptance. Mentor will spend about $6 million less on call center services this year thanks to the SupportNet upgrade, according to Floodeen, yet the total project cost $4 million, including $1 million on the InQuira software and the remainder on Siebel and Primus (now ATG) software upgrades and consulting.

Use What's In Place

The University of Pittsburgh Health Sciences took a federated approach to enable students and medical professionals to search electronic sources not included in its library catalog. These external resources were licensed by the library but maintained on separate servers by various publishers.

"As our digital library grew, the location of sources was becoming scattered," says associate library director Deborah Silverman "Users needed to know where all these resources where before they could even get started." "They shouldn't have to do that."

To aggregate search results from multiple sources, the University early this year deployed Vivisimo's Velocity search engine to perform a federated search of internal resources as well as the third-party publisher sites. To improve accuracy, the issue raised by Reynolds, Velocity uses a clustering technique that organizes search results into categories that are presented to users in a tree format. Users select the appropriate search path to refine their searches and cut down on information overload. The number of search iterations per user has declined since the system was deployed and that is, says Silverman, clear evidence that individual searches have become more effective.

From Awareness To Intelligence

Ultimately, the goal of search is simple: to find answers and make better decisions. Putting people in touch with more sources of information, including content and data as well as internal and external sources, is a step toward information awareness. To get to information intelligence — the ability to find the right information and associations that might not be obvious at first glance — users are experimenting with text mining, data extraction, clustering, guided navigation and other advanced techniques that are helping to make overload a thing of the past.

Michael P. Voelker is the Principal of Equinox Communications. Write to him at [email protected].

OPINION: Enterprise Search Is A Myth

Is enterprise search something that's real and achievable? Or is it a myth-the latest in a long string of hype stretching back to artificial intelligence?

Since the number of webinars, articles, conferences and .ppt files on the subject has exploded over the past 18 months, it's tempting to conclude that we're really getting someplace with enterprise search — this time. Attractive new products and new companies have appeared, including, for example, broad-spectrum search supplier Fast Search & Transfer, X-1 in personal search, Vivissimo in clustering and Attensity in content analytics. The largest of enterprise and desktop software suppliers (Microsoft, Apple, IBM and SAP) have also rediscovered the importance of search. Even the Internet supersites Google and Yahoo! want to contribute, offering desktop search tools and, in the case of Google, server-based search appliances.

Delphi Group Research shows that only about one professional in eight claims access to a search tool that integrates different sources of information (see the "Listening Posts"). Only one in 20 claims to use a single search tool in their work, and one third say they use four or more search tools. When we asked executives how they're addressing these challenges, only one in seven said their firms have an enterprise search strategy.

What emerges clearly in our research is a picture of application-delivered search silos. Each desktop application, each enterprise application, corporate portal and Internet destination helpfully offers a search box. At our own computers, we can now download from Google, Yahoo! and others tools to search our personal hard drives. We will have other versions of local search at the operating system level-available from Apple (Spotlight search in the Tiger OS) and coming soon in Microsoft's Longhorn.

Let's remember that what people want from search in the work environment is fast, accurate answers. As long as search means no more than an invitation to browse through hundreds if not thousands of documents, it's not the answer, and an enterprise-size version won't help.

Tomorrow's best practice for search will lie not in corralling more and more disparate sources into the same broken metaphor. In fact, the best approach won't look like search. It will be knowledge driven-tailored to business processes-and it will take advantage of our rich metadata infrastructures to focus quickly on the context of user problems. Examples of business-context focused applications include Stratify's product designed expressly for the legal discovery environment, Intelliseek's marketing intelligence application (which analyzes consumer opinions and experiences aired in public Internet communities, forums and blogs), and ClearForest's manufacturer-oriented application for spotting product quality and warranty problems.

These new applications take advantage of a spectrum of advanced analytics-from natural language search and dynamic classification to text/data mining, fact extraction and real-time alerting-to provide answers rather than increasingly long document lists.

View the enterprise search challenge not as achieving a single search box, but rather as an opportunity to create high-value enter-prise applications. Based on advanced analytics, these apps will increasingly combine analysis and profiling of unstructured and structured information. Most importantly, the focus will shift to limiting what information gets delivered to users to relevant and actionable materials. The sooner IT can break the choke hold that the one-million-documents-match-your-query syndrome now exerts on productivity, the sooner professionals will want to leave today's "enterprise" search box experience behind.

Hadley Reynolds is research director at Delphi Group. Write to him at [email protected].