Government // Enterprise Architecture
Commentary
9/6/2012
03:38 PM
Paul McDougall
Paul McDougall
Commentary
Connect Directly
RSS
E-Mail
50%
50%

Bing Flunked My Search Taste Test

Microsoft claims Bing's results are preferred over Google's by 2 of every 3 searchers--but, for me at least, the numbers don't add up.

Microsoft SkyDrive Vs. Dropbox, Google: Hands-On
Microsoft SkyDrive Vs. Dropbox, Google: Hands-On
(click image for larger view and for slideshow)
Microsoft this week launched a website that lets users input search queries to see results from Bing and Google side by side, without knowing which is which. Users can then vote for the results they prefer. The exercise is meant to support Microsoft's claim that people looking for information like Bing results over Google by almost two to one. That's not exactly how it turned out for me.

I preferred the Bing results just once in five trials. Google won three times, and I scored one test a draw. Overall, I found that Google's results pointed me more directly to what I wanted, and more frequently offered enhancements, such as maps and deep links.

Let's walk through my results.

I'll be traveling to Canada next week to play ice hockey. Whenever I think hockey, I think of my hometown Toronto Maple Leafs, so "Leafs" was my first query term.

Bing's top result was a link to the team's home page, with deep links to stats, rosters, tickets, and the like. Google also first yielded a link to the home page, but no deep links. Bing's second result was a news story about Leafs' captain Dion Phaneuf getting engaged to "24" star Elisha Cuthbert. Google's second result was a Wikipedia entry explaining what a leaf, as in the non-hockey playing plant variety, is. Google 0, Bing 1.

[ A 2011 study gave Bing, not Google, the edge. Bing Beats Google On Search Effectiveness. ]

Next I searched for my name. Bing yielded an outdated byline photo of me at the top, along with images of four other Paul McDougalls (get me Apple's patent lawyers). Google's first result was a link to PaulMcDougall.com, which has nothing to do with me but promises to help you learn the alphabet "with the help of some charming young creatures and their quirky friends."

Google's second result was my professional bio on InfoWeek.com. Bing's second reference was a Paul McDougall on Storyboard.com, who "lives in a world full of quirky creatures, friendly monsters and oddball ghosts." That is actually all true about me, but wasn't me. Bing's results sample didn't find my bio at all. Google 1, Bing 1.

Back to my hockey trip. I'll be flying to Spokane, Wash., and driving to Invermere, B.C., where the game, an alumni match for a team I played for years ago, is to be held. I input "Spokane to Invermere B.C. driving directions." I figured that was a good test, because although Invermere is a beautiful town in the Kootenay mountains, and is right next to the world-class Panorama ski resort, it's not a household name.

Bing only produced links showing distances between seemingly random points in the western U.S. and Canada. Google's first response was what you want when you need directions--a map. It also gave me the precise distance, 271.4 miles. Google 2, Bing 1.

Next, I'll need a place to stay. So I input "Invermere, B.C. hotels." Bing's first instinct was to display a link to Hotels.com's Invermere page. Google's initial results provided direct links to specific inns and hotels, as well as real-time prices. Much more relevant. Google 3, Bing 1.

For my last test--BingItOn.com gives you five rounds per session--I figured I might need some new equipment for the game. I input the specific make, model, and size of the goal pads I want, "Brian's AirPac 34 inch". Serious hockey players know what that means, but neither Google nor Bing had much of a clue. They both yielded links to outdated eBay auctions, old forum postings, and such. I scored it a tie. Google 3, Bing 1, Draw 1.

So, in five blind tests, Google won 60% of the time, with Bing prevailing just 20% of the time. Now, this is of course subjective. Some people might actually prefer search engines that have no clue who they are. But in most respects I'm a typical searcher. When I need driving directions, I want the results with a map.

So let's put some science behind this. If the chances of a random user like me preferring Bing results over Google were fifty-fifty, then the probability of Bing winning only one in five trials is--using the binomial random variable formula, which yields probabilities for specific outcomes, given an underlying probability--just 15.6%. In other words, it's unlikely Bing was just having an off day.

But there's more. According to Microsoft, Bing was preferred by roughly 66% of users--so it shouldn't be a 50/50 proposition. If two-thirds of users really prefer Bing, the probability of it having only a 20% success rate across a set of five random trials reduces to just 4.4%. Hmmm.

What's Microsoft's point behind the site? The company insists that Google has become so synonymous with search that Bing doesn't get a fair shake. And that's why Bing's share of the market is, according to Comscore, just 15.7%, compared to Google's 66.8% share. All Microsoft is saying is give Bing a chance. You can try the test at BingItOn.com.

Let me know your results.

At this year's InformationWeek 500 Conference, C-level execs will gather to discuss how they're rewriting the old IT rulebook and accelerating business execution. At the St. Regis Monarch Beach, Dana Point, Calif., Sept. 9-11.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 3   >   >>
computererds
50%
50%
computererds,
User Rank: Apprentice
4/19/2013 | 4:03:14 AM
re: Bing Flunked My Search Taste Test
I think your first theory might be more valid. I did 4 natural language searches and one specifically keyword booleen search. I ended up with a score of 4-Google 1-draw.

I also had to re do all the searches in Internet Explorer so I could see the results page. To me that says just as much as the search results.
PMcDougall
50%
50%
PMcDougall,
User Rank: Apprentice
9/10/2012 | 6:58:23 PM
re: Bing Flunked My Search Taste Test
thanks for the interesting feedback BlackCats. I'd argue that the independence criterion is satisfied because it's a blind test--I doubt most people could simply guess which was which.

on the Bernoulli, I went with p=.66 based on Microsoft's statement that "nearly two out of three" prefer Bing. Thanks for pointing out that it's actually a good bit lower than that--.57. (though I did provide another example using p=.50, which still cast doubt on Microsoft's assertion).

Thanks again, cheers.
BlackCatsOnly
50%
50%
BlackCatsOnly,
User Rank: Apprentice
9/10/2012 | 5:33:07 PM
re: Bing Flunked My Search Taste Test
@PMcDougall: There are several problems with this logic.

First, the Bernoulli Process requires independence. It's not clear that can be guaranteed in this experiment because the challenge is largely targeting users who already use Google (because most users are currently Google users.) The user will be trying to confirm his bias, so whatever perception of result characteristics they have is likely to make them go mostly one way or mostly the other. So balanced results where the scores are close (e.g., 3-2 or 2-2-1) could be less likely. Thus, people really latched to one engine or the other so that they somehow really identify with that engine's results would be getting 4's and 5's for their preferred engine. Again, independence of individual trials is not obvious in this scenario.

Second, the study (http://www.bing.com/community/... does NOT say the underlying success rate for each trial is 66%. It actually says "57.4% chose Bing more often, 30.2% chose Google more often; 12.4 % resulted in a draw." These results are reporting the outcomes of the full experiments (5 searches), NOT the outcomes of the underlying trials. So, for the Bernoulli Formula, p is not known to be .66 (especially in the context of the dubious independence of the trials, noted above.)

So to expand on my previous summary, to determine the "p" that should be used in the formula, you'd need to resolve concerns about independence. Getting access to the actual underlying trial result data would help a lot.

To be sure, it could be that the underlying percentages for the individual trials look a lot like the percentages for the experiment outcomes, but that is not at all obvious to me.
Andrew Hornback
50%
50%
Andrew Hornback,
User Rank: Apprentice
9/8/2012 | 9:17:38 PM
re: Bing Flunked My Search Taste Test
Paul - I think I can answer why Google won out over Bing for you.

I highly doubt that you're the average Internet consumer. That said, you've got years of experience using tools on the 'net to find the information that you're looking for. Corollary to that, you know how to craft a search in order to find what you're looking for faster.

That said, I decided I'd take up the test and try it out myself. Bing 1, Google 1, Draw 3.

I ran through the test the following search terms:

Willie J. Laws - draw
black molle - Bing, more shopping oriented
dodge magnum camshaft - draw
L-Theanine - draw
Corporation Blue - Google (Bing was coming up with results that included corporation or blue, but not both, oddly enough and neither necessarily had what I was looking for)

I wonder who in Microsoft's marketing organization came up with this idea... and if they'd be interested in a bridge I'm selling down in Brooklyn, for cheap.

Andrew Hornback
InformationWeek Contributor
sm5669
50%
50%
sm5669,
User Rank: Apprentice
9/8/2012 | 3:21:52 PM
re: Bing Flunked My Search Taste Test
I took the Bing it On test 4 times with Google coming out on top all 4 times.
Twice Google won 4-1 and twice Google on 5-0.
I mixed up my searches between automotive repair, home repair, entertainment trivia, local business searches, sports and technology topics.
In many cases the results were very similar with the difference being in how the results were formatted/displayed.
moarsauce123
50%
50%
moarsauce123,
User Rank: Ninja
9/8/2012 | 1:16:30 PM
re: Bing Flunked My Search Taste Test
Tried this and Google reported the distance as 271.4 miles versus Bing's 271.0 miles. Googles driving time was 5 h 33 m, for Bing it is 5 h 4 m. Location was the same for both and Google showed me the map as part of the search results without having to click on anything. Also, Google Maps uses much more current map material than Bing. Bing's maps are often horribly outdated.
So what's your point? The .4 mile difference? And with Google I do not have to interact at all, because the result was right there in the top spot. And Google was noticeably faster, because it doesn't have the fancy background graphics and all the other useless garbage that Bing barfs onto the screen. I'm willing to call this a tie, but one where Google is easier to use and faster.
moarsauce123
50%
50%
moarsauce123,
User Rank: Ninja
9/8/2012 | 1:08:24 PM
re: Bing Flunked My Search Taste Test
Flunked for me as well, Google 4 Draw 1. Microsoft should take that site down before it gets to embarrassing.
ZoZa63
50%
50%
ZoZa63,
User Rank: Apprentice
9/7/2012 | 10:10:21 PM
re: Bing Flunked My Search Taste Test
Google 5 Bing 0
Maybe it's because its my default search and I'm more familiar with Google?
awebb199
50%
50%
awebb199,
User Rank: Strategist
9/7/2012 | 10:00:25 PM
re: Bing Flunked My Search Taste Test
Bing 3 times, google twice. I use Bing almost all the time.
PMcDougall
50%
50%
PMcDougall,
User Rank: Apprentice
9/7/2012 | 7:01:24 PM
re: Bing Flunked My Search Taste Test
@BlackCats. You stated: "Summary: without access to the actual data, the user preference stat says nothing about the likelihood of a single query going one way or the other."

Just to be clear, that's not what I said in the story. What I said was the odds of Bing winning only one of five trials (doesn't matter which one), is 4.4%, if Bing really does have an overall success rate of 66%, as claimed by Microsoft.
Page 1 / 3   >   >>
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Must Reads Oct. 21, 2014
InformationWeek's new Must Reads is a compendium of our best recent coverage of digital strategy. Learn why you should learn to embrace DevOps, how to avoid roadblocks for digital projects, what the five steps to API management are, and more.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
A roundup of the top stories and community news at InformationWeek.com.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.