Credibility Of Analysts

Bias is a nonissue at the company, CEO Hall insists. "We wouldn't have a dollar of revenue from the user community if our objectivity and independence weren't held in high regard," he says. Gartner has policies in place meant to ensure objectivity. The ombudsman office reports to Gartner's general counsel to ensure it's free from pressure from other parts of the company. And Gartner analysts aren't allowed to own stock in the companies they cover.

Execs at other major research firms speak with similar conviction. "We provide fact-based advice," says IDC CEO Kirk Campbell. IDC's research methodology, with its emphasis on hard data, provides a built-in guard against analyst bias or favoritism, he argues. Campbell dismisses any suggestion that vendors have to be paying customers to get fair treatment in IDC reports. "We have an open-door policy," he says.

It sounds convincing, but there's an important bit of information that Campbell refuses to share: He won't disclose how much money IDC takes from tech-vendor clients.

Forrester Research focuses on selling its services to users rather than vendors to ensure that most of its revenue doesn't come from the subjects of its research, says Brian Kardon, chief strategy and marketing officer. That affords the company a lot of freedom. "We routinely slam vendors," he says. Still, about a third of Forrester's revenue comes from tech vendors.

Some research execs concede that their firms must do a better job of educating customers and the public about their policies and procedures, given the influence they have over multimillion-dollar buying decisions. One idea is to develop industrywide standards for business practices. "It's something we should look at," says Yankee Group CEO Emily Green. "Even the perception of favoritism would hurt us." Says Forrester's Kardon, "It would help the whole industry if we had a common set of practices to keep everybody clean."

Aberdeen Group CEO Jamie Bedard last week claimed the high ground. In a progress report on Aberdeen's business, Bedard wrote to customers, "We promised you that ... our research integrity was not for sale." In an interview, Bedard charged that too many research firms base their advice on just a few interviews--what he calls opinion-based research--rather than on detailed surveys of dozens or hundreds of companies. "I think the industry can do a better job of deep research," he says.

On The Radar

The stakes for technology vendors can't be underestimated. Certain vertical industries and large companies look to Gartner for guidance on which products they should include in a technology bake-off, says Vikram Phatak, CTO of intrusion-prevention system maker Lucid Security, a Gartner client since March. This means Lucid must be on Gartner's radar screen when the analyst firm meets with customers who want advice about the IPS market. Phatak expects Lucid to go public or be bought within the next 18 months, so being in the visionary sector of Gartner's IPS Magic Quadrant is crucial. "If you're in the Quadrant, you're a safer bet to potential buyers and VCs than if you're not," he says.