Web Search, Transformed: Looking Out For You

For many of us, "search" is still a matter of typing words into a field and waiting a few seconds for web results.

And it's so 1999.

"People have been using the web for search the same way for 15 years despite that fact that the structure of the web has fundamentally changed," says Stefan Weitz, director of search at Microsoft and author of a new book, Search: How The Data Explosion Makes Us Smarter.

But there's a transition underway in search through the increased use of voice assistants like Siri, Internet of Things sensors, and data analytics tools that are changing "search" from something you do to something that happens for you based on your queries, your location, and what you're doing in the virtual and physical worlds, says Weitz.

This is happening as the web itself transitions from a repository of text and images to a "tool that transports data and allows things to happen in real life."

When you try to go beyond the text-based search we've come to know, says Weitz, the limitations become clear.

[Google's experimental email client Inbox will change how you look at email. Read Google Inbox: 7 Reasons To Say Yes]

"If I type in a phrase like 'get a reservation tonight at 7 p.m. at Wild Ginger,' traditional search will, at best, return a set of pages from which I could take action, but that's not what I want," he writes in Search.

"I want to complete my task without intermediary steps. I'd like the engine to know who I am, to know what I likely mean, and to find out if there is a table open tonight. If there's not, I'd like it to recommend a restaurant that I might like instead, based on my request and what it knows about me. But today's search gives us links to web pages because the engines still treat all queries as requests for documents, images, or video."

Search explores what happens when web search starts treating the web as a digital stand-in for the physical world. With increasingly cheap computing capacity via the cloud, an abundance of data-gathering sensors, artificial intelligence systems like Siri, Microsoft Cortana, and Google Now, and the troves of digital data piped to the web, search engines are becoming "a watchful eye that you've told to do and predict things for you."

A "watchful eye" is an unsettling concept for many people in light of revelations about NSA spying programs and data breaches at Target and Home Depot. But for Weitz, it brings to mind the age-old privacy trade-off: Is the reward you get worth the personal information you've allowed the service to have on you?

"If I'm going to let you have my exact location all day and I get a car dispatched to me in two minutes, then that's worth it to me," said Weitz in a phone interview with InformationWeek. "But if I'm just targeted for ads about soccer balls? Not worth it."

Weitz is optimistic about search engines making us smarter and more productive -- he is, after all, in the search business. (It's worth noting that the book is decidedly vendor-neutral despite Weitz's employment at Microsoft.) His contention is that search advances will simplify our lives, help us make quality decisions, and steer us away from actions that do not benefit us -- such as that flight you're going to miss or that gridlocked traffic awaiting you.

However, the book doesn't shy away from discussing the substantial challenges of reaching search omniscience, such as: the overwhelming costs -- in the many billions -- of building and operating search systems that have long relied on advertising to generate revenue; the costs of processing mountains of data; having data silos that are not shared across services. (Your Amazon data, your Netflix data, your credit card data, and so on, cannot speak to each other, but allowing them to do so would greatly improve search, Weitz says.) And, of course, there are the concerns over privacy violations.

"We've never had so much information about humanity in a digital accessible form, and it's a profound change," he says. "Search engines are building a digital avatar of you and there are privacy challenges for sure. You have to ask: who owns the data and how is it being used? I expect regulations to be actively put in place across the industry to protect personal data, but the concerns are valid."

Better search, in the end, hinges on our willingness to digitize all our data, says Weitz, and while he acknowledges that digitizing everything about ourselves is not everyone's cup of tea, it's the only way "to see the truths hiding in the data that we, as humans, can't process."

The onus, he says, is on the tech community to help people answer the question of why they should let systems have access to personal data. They need a compelling reason. "It wasn't until there was a clear benefit to volunteering personal information that people -- tens of millions of them -- bought fitness-tracking wearables like Fitbit, Nike+ FuelBand, and Jawbone Up," Weitz writes in Search.

Whether or not you agree with Weitz's "digitize everything" vision, Search is a thought-provoking assessment of the agony and ecstasy in reaching the lofty goal of ubiquitous search, and is well worth reading.

Apply now for the 2015 InformationWeek Elite 100, which recognizes the most innovative users of technology to advance a company's business goals. Winners will be recognized at the InformationWeek Conference, April 27-28, 2015, at the Mandalay Bay in Las Vegas. Application period ends Jan. 16, 2015.