Back in seafaring days of yore, ship captains with unsavory crews made a habit of adding all sorts of phony steps to the process of using the sextant, compass, watch, and chart, lest potentially mutinous hands catch wise to the otherwise straightforward science of navigation.

They'd take multiple sextant readings of no consequence, swing their pocket watch like a pendulum, counting meaningless beats, then scratch down nonsensical calculations. The show made the task of choosing a safe course seem like a mysterious art that only the learned captain could handle.

Flash forward a few centuries and consider the Gartner Magic Quadrant, the analyst firm's popular charting of IT vendor leadership. How does Gartner go about triangulating the vendor dots on the horizontal "completeness of vision" and vertical "ability to execute" axes? Detractors have taken their shots at the process over the years, comparing the Magic Quadrant to a Magic 8 Ball or dartboard, even accusing Gartner of favoring "whoever pays the most."

Gartner last week published its business intelligence (BI) and data warehousing (DW) quadrants, and I have to break with the pundit tendency to bash these reports. In fact, I'm not the only observer who has noticed a bit less dark art and a bit more transparency in Gartner's process--at least where the BI and DW reports are concerned.

[ Want more on business intelligence and the data platform? Read our 2012 BI & Information Management Report. ]

Gartner uses a well established methodology (explained in this FAQ) across all quadrants. It entails a process of (typically annual) vendor briefings and surveys followed by customer surveys and interviews. The teams of analysts who write up each report stick to the general methodology, but they also develop market- and product-specific criteria, and they're free to put a greater emphasis on, say, the customer or vendor surveys, depending on the maturity of the market in question.

Gartner's BI Quadrant, which is available through this Tableau Software link, has relied much more heavily on customer surveys in recent years. Gartner surveyed 1,364 customers this year, and its findings were applied to three out of seven criteria behind the vertical "ability to execute" axis (namely, assessments of the product/service, sales execution/pricing, and customer experience).

Over the last three years, Gartner has also published the results of the customer survey, giving practitioners and vendors alike a chance to read verbatim quotes and scoring results. "We think the [Quadrant] process is definitely becoming more transparent over time, which is good for all the vendors and the customers who rely on this information," says Dan Jewett, VP of product management at Tableau Software, one of two "challengers" in this year's BI report (along with Tibco Spotfire).

The survey write-ups are particularly useful, says Jewett, because tech buyers can see how different vendors stack up on the attributes they're most concerned about.

Some vendors aren't so keen on Gartner's customer survey approach. SAP, for one, was somewhat dismissive of this year's BI quadrant as "a subjective ranking system with a heavy weighting on a small sample of customer survey responses," as described in a statement by Jason Rose, SAP's VP of business intelligence marketing.

What's surprising is that SAP is one of eight companies Gartner placed in the BI leaders' quadrant. But the company had the lowest "ability to execute" among those eight, ranking below Oracle, MicroStrategy, Microsoft, Information Builders, QlikTech, IBM, and SAS, in that order.

I, for one, don't consider 1,364 to be a small sample, especially considering that Gartner's sample two years ago included fewer than 900 respondents, including 91 SAP BusinessObjects customers. SAP's dot was in a similar spot back then.

Data Warehouse Vendor Rankings

Gartner's DW quadrant is far less customer-survey-intensive than the BI quadrant, acknowledges co-author/analyst Donald Feinberg, but he says he and his peers are using more survey data than ever before. "We've not only encouraged vendors to send us more references, but we're sending more surveys to Gartner clients," he explains.

There weren't any real shocks in this year's DW leaders quadrant. Teradata, Oracle, and IBM, respectively, still lead the rankings. EMC and Sybase swapped positions (roughly), while Microsoft somehow slipped into leader's quadrant, despite the fact that Gartner says it has yet to see a production deployment of Microsoft's SQL Server Parallel Data Warehouse. (This last case is one of those instances where you sense a little "magic" at work.)

What some report readers find most puzzling about Gartner's quadrants is the fact that vendors never reach the upper-right-hand corner. In fact, Teradata, Oracle, and IBM are further from that corner this year than they were last year. This evolution, says Feinberg, simply reflects the fact that markets are dynamic and that expectations keep rising. One year ago, big data was of growing importance, but customers now expect a comprehensive vendor vision and ability to execute on that score.

"A lot of people make the mistake of thinking of the edges of the quadrant as fixed, when actually it's the center that's fixed," observes Tim Negris, VP of marketing at cloud-based data warehousing vendor 1010data, the lone "challenger" in this year's Gartner DW Magic Quadrant, which is available through this 1010data link.

In other words, the average vendor vision on the horizontal axis and average customer expectations for ability to execute are dead center, but that's a moving target, explains Negris, who has also briefed Gartner analysts when he worked for Oracle and IBM.

None of this scrutiny would matter if nobody paid much attention to Gartner's quadrants, but it's not just vendors looking in the mirror, Negris says. "Judging by the standing-room-only crowds at Gartner conferences when they go through these reports in detail, customers are very interested because it helps them see what's happening," he says.

My point is that there's a lot of rigor behind the quadrants, and the BI and DW reports that I cover have only improved with the inclusion of deeper customer research. I know from first-hand experience that our own InformationWeek assessments, like our 2012 BI & Information Management Report, hugely benefit from in-depth end-user surveys.

As one critic of the recent Forrester Wave Report on Enterprise Hadoop Solutions advised, even if you take exception to the ranking of this or that vendor, you can get a lot of value just by looking past the chart and reading the content of the report.