The Real Reasons Enterprise Search is Broken

I had one of those midnight "wake up and go Doh!" moments last week. A common feature across nearly every conversation I've had about Enterprise 2.0 subjects hit me. Everybody says "Enterprise search is broken." In fact it is one of the first things to come up. But then people move on. As Churchill once said, people often stumble across the truth, but most pick themselves up and move on. I am guilty too. I first "stumbled" 3 years ago, and it's taken me this long to say, "wait a minute, I never thought that through."People move on because they seem to assume that this is incompetence at work. Search is sooo 1.0, right? It's been solved, and we're just fumbling the execution, right? You usually get some sort of ironic joke along the lines of "wow, it is so easy to find stuff out there on the public Web, and here with all our resources, we can't even do search right."And then the conversation tends to move on to more obviously "2.0" things like blogs, wikis, how to increase participation, and my personal pet peeve: annoying moaning about "culture change."Hold on. Rewind. Let's go back to search and think for a moment. I have a theory here, and I'd like to see if all you smart E 2.0 guys agree. I have reached a radical conclusion: broken search is the problem, but fixing search is not the solution. Search breaks behind the firewall for social, not technical reasons.How Search Breaks Behind the FirewallLet's start with the blindingly obvious, and then draw some weird conclusions. Here are the most common reasons that come up:

Sounds like a purely technical problem doesn't it? Get it all SOAn up (apologies for the terrible pun), create the holy grail "email attachment killing" technology (or alternately, figure out a way to make email "socially searchable").Why Search Breaks Behind the FirewallOkay, I've been leading you to what I believe is the wrong solution. Everything above is merely a symptom. You could put the most talented engineers onto the problem, spend billions, and still end up with a mess.You see the problem is not technical. It is social.The key here is all those "ought" phrases. Who's recommending the "ought"? Who is being recommended as the "ought to do something" constituency? There's no such thing as a person or organization without an agenda within a corporation. If the IT czar is a psycho with crazy ambitions, do you really want him/her to have all the skeleton keys? Is the person saying "ought" things about email attachments a librarian trying to make his/her job easier? Or somebody who feels cut out of certain conversations? Or somebody who is genuinely pointing out a non-political problem with sloppy sharing?We often forget that the public Web is a very democratic place. Authority is democratically earned. PageRank is democratic. We link to stuff we like, authors we like. Google's algorithms mine a vast democratic vote. Stuff on the public Web is online primarily because people WANT it online. They WANT it to bubble up. There are no big devious intentions. On the public Web, popularity equals power in very direct and simple ways.Now step behind the firewall. What do you see?

Forget blogs, forget wikis. Quite often intranet bloggers and wiki champions are the zero-influence noise signals (and I say this despite being one myself!). Most of the powerful people won't blog. Let's say a bright young intern writes a killer strategy brief/analysis of the company's positioning as an internal blog, which gets a lot of comments and chatter. Sure, an alert VP might spot it and hire the kid and groom him/her (okay I'll admit it, that happened to me!). But look at what that does to search. That page might well top the rankings by normal "democratic" search logic for the word "strategy," but truth be told, the most important strategy documents are probably some low-distribution emails/meeting minutes, or some very carefully judged piece of officialese that comes out as a fait accompli, when it is useless to everybody else. Most people take one look at it, and yawn, but it is designed to send the right coded, and tactically timed signals about who has what piece of turf, to the few people who can interpret it. This document may get buried.There are even rules. Like actual government, go-to-jail rules, that prevent the most important stuff from being shared at all beyond corporate officers, let alone discussed.Some of the polyannish types might claim that this implies the organization is broken and that there is a need for more transparent, open, democratic governance.I say, HELL no! This stuff may sound horrible, but large companies are not democratic for a reason. They exist to make money, not as laboratories for empowerment or social engineering. The authority structures, communication flow patterns, emergent models of what's hidden and how and why, versus what's open, are all designed to work. Sure, you can find a lot of alternative models that are starting to work, but let's not throw out the old without understanding how and why it works.Which brings me to my main point. Search is political.Search is PoliticalWe often forget that Web 2.0 is built on top of Web 1.0. It assumes functioning search as part of the canvas. Google and other search majors attempt strongly to maintain a "search neutrality" that matches "Net neutrality." You may not like to hear this, but "search neutrality" is a really bad idea inside the firewall.Inside the organization, the main form of authority and control is based on patterns of information sharing. We assume that function is accomplished by permission controls on content management systems and the like. Not true. Most information flow control is not based on restricting access, but limiting the very knowledge that a certain piece of information even exists. And this is a good thing. If a lunatic in some toxic silo has evil designs on another department, then even the knowledge that a certain document exists may need to be hidden from him/her. There may be very good tactical reasons for burying some information in a complicated email thread, rather than being shared with clarity. Harsh but true: there are many good things that get done because someone was smart enough to cover their actions with nominal disclosure ("but it was in that email I sent 3 weeks ago, don't act surprised.")Yes, such toxic politics can be really bad for an organization, but the solution is not to deny that it exists, but be smart about managing it. Ben Horowitz has a brilliant recent post about what to do about minimizing politics from the CEO point of view, but it is really everybody's job to be sophisticated about this stuff. And shoving a democratic search engine into your information ecosystem is a pretty naive thing to do.A large organization is a contentious place. Alignment isn't some idealized state of harmony to aspire to. It is a decision-by-decision evolving reality as power and influence shift through the organization chart. And again, this is a good thing. If things are going badly, you want different organizations to debate and express dissent, negotiate and if necessary, yes, hide information. You might even want them to work against each other for a bit of fairly-structured internal competition if you are not sure who has it right.Public Web models of search blunder into this intricate and complex piece of machinery like clueless bulls in a china shop. This is the reason why they very rarely get much traction. It is in almost nobody's interest to have powerful enterprise-wide search in place. This may sound perverse, but the ability to penetrate the opacity of organizations is to a certain extent the right test of whether you deserve access at all. If you can get to it, you are smart enough to use it wisely. It's not a great system, but it's better than naive search.Instead of SearchI've been thinking hard about this problem, and I don't think enterprise search can or should be "fixed." The fundamental social and information flow assumptions of "search" need to be deconstructed and reconstructed for the enterprise. Local/silo search within single sites/assets is fine. Enterprise-wide search in its naive form is a terrible idea. And it is actually a good thing it is being done badly. Doing enterprise search well will provide a LOT more visibility to the most useless parts of the information ecosystem within an enterprise, and provide more incentives to hide the important information. Search infrastructure can provide a false sense of security, openness, transparency and "being informed."What can we do instead of search? I have some ideas, but they are still at a very early stage. But I do know that a smart discovery function is essential before the rest of the E 2.0 suite can transform organizations. The Web 1.0 agenda is still unfinished within the enterprise, so let's be cautious about moving on to the E 2.0 agenda. Plumbing before furniture.I am curious about how many people agree with my diagnosis or have alternative ideas/mental models here. Apologies if I've offended any huge advocates of enterprise search.(Anyone know of any interesting forums/email lists about this issue? If not, I might be interested in starting one. Email me if you are interested, and I'll start some sort of Google group or something if there's enough interest).Venkatesh Rao is an Entrepreneur-in-Residence at Xerox, and the manager of the Trailmeme project. He blogs at Ribbonfarm.com. These views are his own, and do not reflect those of his employer.