Bing Flunked My Search Taste Test

Microsoft SkyDrive Vs. Dropbox, Google: Hands-On

Microsoft this week launched a website that lets users input search queries to see results from Bing and Google side by side, without knowing which is which. Users can then vote for the results they prefer. The exercise is meant to support Microsoft's claim that people looking for information like Bing results over Google by almost two to one. That's not exactly how it turned out for me.

I preferred the Bing results just once in five trials. Google won three times, and I scored one test a draw. Overall, I found that Google's results pointed me more directly to what I wanted, and more frequently offered enhancements, such as maps and deep links.

Let's walk through my results.

I'll be traveling to Canada next week to play ice hockey. Whenever I think hockey, I think of my hometown Toronto Maple Leafs, so "Leafs" was my first query term.

Bing's top result was a link to the team's home page, with deep links to stats, rosters, tickets, and the like. Google also first yielded a link to the home page, but no deep links. Bing's second result was a news story about Leafs' captain Dion Phaneuf getting engaged to "24" star Elisha Cuthbert. Google's second result was a Wikipedia entry explaining what a leaf, as in the non-hockey playing plant variety, is. Google 0, Bing 1.

[ A 2011 study gave Bing, not Google, the edge. Bing Beats Google On Search Effectiveness. ]

Next I searched for my name. Bing yielded an outdated byline photo of me at the top, along with images of four other Paul McDougalls (get me Apple's patent lawyers). Google's first result was a link to PaulMcDougall.com, which has nothing to do with me but promises to help you learn the alphabet "with the help of some charming young creatures and their quirky friends."

Google's second result was my professional bio on InfoWeek.com. Bing's second reference was a Paul McDougall on Storyboard.com, who "lives in a world full of quirky creatures, friendly monsters and oddball ghosts." That is actually all true about me, but wasn't me. Bing's results sample didn't find my bio at all. Google 1, Bing 1.

Back to my hockey trip. I'll be flying to Spokane, Wash., and driving to Invermere, B.C., where the game, an alumni match for a team I played for years ago, is to be held. I input "Spokane to Invermere B.C. driving directions." I figured that was a good test, because although Invermere is a beautiful town in the Kootenay mountains, and is right next to the world-class Panorama ski resort, it's not a household name.

Bing only produced links showing distances between seemingly random points in the western U.S. and Canada. Google's first response was what you want when you need directions--a map. It also gave me the precise distance, 271.4 miles. Google 2, Bing 1.

Next, I'll need a place to stay. So I input "Invermere, B.C. hotels." Bing's first instinct was to display a link to Hotels.com's Invermere page. Google's initial results provided direct links to specific inns and hotels, as well as real-time prices. Much more relevant. Google 3, Bing 1.

For my last test--BingItOn.com gives you five rounds per session--I figured I might need some new equipment for the game. I input the specific make, model, and size of the goal pads I want, "Brian's AirPac 34 inch". Serious hockey players know what that means, but neither Google nor Bing had much of a clue. They both yielded links to outdated eBay auctions, old forum postings, and such. I scored it a tie. Google 3, Bing 1, Draw 1.

So, in five blind tests, Google won 60% of the time, with Bing prevailing just 20% of the time. Now, this is of course subjective. Some people might actually prefer search engines that have no clue who they are. But in most respects I'm a typical searcher. When I need driving directions, I want the results with a map.

So let's put some science behind this. If the chances of a random user like me preferring Bing results over Google were fifty-fifty, then the probability of Bing winning only one in five trials is--using the binomial random variable formula, which yields probabilities for specific outcomes, given an underlying probability--just 15.6%. In other words, it's unlikely Bing was just having an off day.

But there's more. According to Microsoft, Bing was preferred by roughly 66% of users--so it shouldn't be a 50/50 proposition. If two-thirds of users really prefer Bing, the probability of it having only a 20% success rate across a set of five random trials reduces to just 4.4%. Hmmm.

What's Microsoft's point behind the site? The company insists that Google has become so synonymous with search that Bing doesn't get a fair shake. And that's why Bing's share of the market is, according to Comscore, just 15.7%, compared to Google's 66.8% share. All Microsoft is saying is give Bing a chance. You can try the test at BingItOn.com.

Let me know your results.

At this year's InformationWeek 500 Conference, C-level execs will gather to discuss how they're rewriting the old IT rulebook and accelerating business execution. At the St. Regis Monarch Beach, Dana Point, Calif., Sept. 9-11.