06:19 PM
Venkatesh Rao
Venkatesh Rao
Connect Directly

The Real Reasons Enterprise Search is Broken

I had one of those midnight "wake up and go Doh!" moments last week. A common feature across nearly every conversation I've had about Enterprise 2.0 subjects hit me. Everybody says "Enterprise search is broken." In fact it is one of the first things to come up. But then people move on. As Churchill once said, people often stumble across the truth, but most pick themselves up and move on. I am guilty too. I first "stumbled" 3 years ago, and it's taken me this long to say, "wait a minute, I never thought that through."People move on because they seem to assume that this is incompetence at work. Search is sooo 1.0, right? It's been solved, and we're just fumbling the execution, right? You usually get some sort of ironic joke along the lines of "wow, it is so easy to find stuff out there on the public Web, and here with all our resources, we can't even do search right."And then the conversation tends to move on to more obviously "2.0" things like blogs, wikis, how to increase participation, and my personal pet peeve: annoying moaning about "culture change."Hold on. Rewind. Let's go back to search and think for a moment. I have a theory here, and I'd like to see if all you smart E 2.0 guys agree. I have reached a radical conclusion: broken search is the problem, but fixing search is not the solution. Search breaks behind the firewall for social, not technical reasons.How Search Breaks Behind the FirewallLet's start with the blindingly obvious, and then draw some weird conclusions. Here are the most common reasons that come up:

  1. Icebergs: Too much stuff that "ought to be shared" is on people's desktops, or sitting in emails in attachment form.
  2. Digital Landfills: There's still tons of those random fileservers all over the place, where people simply dump files. They "ought" to put it into content management systems.
  3. It ain't a net: Web-based content is just easier to crawl. You have all those nice, friendly sociable things like robots.txt files that tell crawlers where to go and not go. Intranets are much ruder. It's hard to take a proper inventory of everything that's not on the searchable "grid" even it is nominally "online" in some form. Corporate IT "ought" to clamp down on undisciplined asset sprawl.
  4. Those pesky formats: PDFs, PPTs, Excel spreadsheets, random proprietary databases with query interfaces.
  5. Permissions: Lots of need-to-know systems with non single-sign-on protections. No way you can give your friendly Intranet spider ALL the skeleton keys, right? Corporate IT "ought" to enforce corporation-wide single-sign on.
Sounds like a purely technical problem doesn't it? Get it all SOAn up (apologies for the terrible pun), create the holy grail "email attachment killing" technology (or alternately, figure out a way to make email "socially searchable").Why Search Breaks Behind the FirewallOkay, I've been leading you to what I believe is the wrong solution. Everything above is merely a symptom. You could put the most talented engineers onto the problem, spend billions, and still end up with a mess.You see the problem is not technical. It is social.The key here is all those "ought" phrases. Who's recommending the "ought"? Who is being recommended as the "ought to do something" constituency? There's no such thing as a person or organization without an agenda within a corporation. If the IT czar is a psycho with crazy ambitions, do you really want him/her to have all the skeleton keys? Is the person saying "ought" things about email attachments a librarian trying to make his/her job easier? Or somebody who feels cut out of certain conversations? Or somebody who is genuinely pointing out a non-political problem with sloppy sharing?We often forget that the public Web is a very democratic place. Authority is democratically earned. PageRank is democratic. We link to stuff we like, authors we like. Google's algorithms mine a vast democratic vote. Stuff on the public Web is online primarily because people WANT it online. They WANT it to bubble up. There are no big devious intentions. On the public Web, popularity equals power in very direct and simple ways.Now step behind the firewall. What do you see?
  1. Every single email ends up being an act of political judgment. To, cc and Bcc are three words about which I could write an entire book.
  2. Need to know and organization charts/cascade patterns beat democratic content popularity and "Word of Mouth" information travel hands down. This surprises people: we often remark about how much work gets done at the watercooler. But this does not mean more gets done that way than through formal channels. The formal channels carry 80% of the communication. The reason we focus on the watercooler is that most of the politically sensitive stuff travels that way.
  3. On the Web, people read people they like. Inside the firewall, everybody is constantly trying to figure out who's important, who has the money, whose stock is up, whose stock is down. Whose coat-tails to ride. Which fool to suffer gladly for the moment, which battle to pick. In other words there is very low correlation between information flows and friendship networks. You may pay most attention to people you hate or fear. Your best friend may be your lunch buddy from another department, and while you may be on each other's blogrolls outside work, you have no connection inside and don't vote up each other's information contributions. Oneworkgroup may be trying to stay under the radar in skunkworks mode, while another may have a reason for wanting a dog-and-pony influence roadshow this year.
Forget blogs, forget wikis. Quite often intranet bloggers and wiki champions are the zero-influence noise signals (and I say this despite being one myself!). Most of the powerful people won't blog. Let's say a bright young intern writes a killer strategy brief/analysis of the company's positioning as an internal blog, which gets a lot of comments and chatter. Sure, an alert VP might spot it and hire the kid and groom him/her (okay I'll admit it, that happened to me!). But look at what that does to search. That page might well top the rankings by normal "democratic" search logic for the word "strategy," but truth be told, the most important strategy documents are probably some low-distribution emails/meeting minutes, or some very carefully judged piece of officialese that comes out as a fait accompli, when it is useless to everybody else. Most people take one look at it, and yawn, but it is designed to send the right coded, and tactically timed signals about who has what piece of turf, to the few people who can interpret it. This document may get buried.There are even rules. Like actual government, go-to-jail rules, that prevent the most important stuff from being shared at all beyond corporate officers, let alone discussed.Some of the polyannish types might claim that this implies the organization is broken and that there is a need for more transparent, open, democratic governance.I say, HELL no! This stuff may sound horrible, but large companies are not democratic for a reason. They exist to make money, not as laboratories for empowerment or social engineering. The authority structures, communication flow patterns, emergent models of what's hidden and how and why, versus what's open, are all designed to work. Sure, you can find a lot of alternative models that are starting to work, but let's not throw out the old without understanding how and why it works.Which brings me to my main point. Search is political.Search is PoliticalWe often forget that Web 2.0 is built on top of Web 1.0. It assumes functioning search as part of the canvas. Google and other search majors attempt strongly to maintain a "search neutrality" that matches "Net neutrality." You may not like to hear this, but "search neutrality" is a really bad idea inside the firewall.Inside the organization, the main form of authority and control is based on patterns of information sharing. We assume that function is accomplished by permission controls on content management systems and the like. Not true. Most information flow control is not based on restricting access, but limiting the very knowledge that a certain piece of information even exists. And this is a good thing. If a lunatic in some toxic silo has evil designs on another department, then even the knowledge that a certain document exists may need to be hidden from him/her. There may be very good tactical reasons for burying some information in a complicated email thread, rather than being shared with clarity. Harsh but true: there are many good things that get done because someone was smart enough to cover their actions with nominal disclosure ("but it was in that email I sent 3 weeks ago, don't act surprised.")Yes, such toxic politics can be really bad for an organization, but the solution is not to deny that it exists, but be smart about managing it. Ben Horowitz has a brilliant recent post about what to do about minimizing politics from the CEO point of view, but it is really everybody's job to be sophisticated about this stuff. And shoving a democratic search engine into your information ecosystem is a pretty naive thing to do.A large organization is a contentious place. Alignment isn't some idealized state of harmony to aspire to. It is a decision-by-decision evolving reality as power and influence shift through the organization chart. And again, this is a good thing. If things are going badly, you want different organizations to debate and express dissent, negotiate and if necessary, yes, hide information. You might even want them to work against each other for a bit of fairly-structured internal competition if you are not sure who has it right.Public Web models of search blunder into this intricate and complex piece of machinery like clueless bulls in a china shop. This is the reason why they very rarely get much traction. It is in almost nobody's interest to have powerful enterprise-wide search in place. This may sound perverse, but the ability to penetrate the opacity of organizations is to a certain extent the right test of whether you deserve access at all. If you can get to it, you are smart enough to use it wisely. It's not a great system, but it's better than naive search.Instead of SearchI've been thinking hard about this problem, and I don't think enterprise search can or should be "fixed." The fundamental social and information flow assumptions of "search" need to be deconstructed and reconstructed for the enterprise. Local/silo search within single sites/assets is fine. Enterprise-wide search in its naive form is a terrible idea. And it is actually a good thing it is being done badly. Doing enterprise search well will provide a LOT more visibility to the most useless parts of the information ecosystem within an enterprise, and provide more incentives to hide the important information. Search infrastructure can provide a false sense of security, openness, transparency and "being informed."What can we do instead of search? I have some ideas, but they are still at a very early stage. But I do know that a smart discovery function is essential before the rest of the E 2.0 suite can transform organizations. The Web 1.0 agenda is still unfinished within the enterprise, so let's be cautious about moving on to the E 2.0 agenda. Plumbing before furniture.I am curious about how many people agree with my diagnosis or have alternative ideas/mental models here. Apologies if I've offended any huge advocates of enterprise search.(Anyone know of any interesting forums/email lists about this issue? If not, I might be interested in starting one. Email me if you are interested, and I'll start some sort of Google group or something if there's enough interest).Venkatesh Rao is an Entrepreneur-in-Residence at Xerox, and the manager of the Trailmeme project. He blogs at These views are his own, and do not reflect those of his employer.

Comment  | 
Print  | 
More Insights
The Business of Going Digital
The Business of Going Digital
Digital business isn't about changing code; it's about changing what legacy sales, distribution, customer service, and product groups do in the new digital age. It's about bringing big data analytics, mobile, social, marketing automation, cloud computing, and the app economy together to launch new products and services. We're seeing new titles in this digital revolution, new responsibilities, new business models, and major shifts in technology spending.
Register for InformationWeek Newsletters
White Papers
Current Issue
Flash Poll
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join InformationWeek’s Lorna Garey and Mike Healey, president of Yeoman Technology Group, an engineering and research firm focused on maximizing technology investments, to discuss the right way to go digital.
Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.