Credibility Of Analysts

Do tech vendors wield influence over IT research? You bet--but how much of it is a matter of perspective?

Research firms make their living by offering expert advice to business and technology people about the best ways to invest their IT dollars. It can be invaluable insight, but only if that analysis comes with no strings attached. And on that, there's no guarantee.

Forrester, Gartner, IDC, and others insist their output is squeaky clean, yet they also rake in millions providing services to the very same companies they monitor, heavyweights like Cisco, IBM, Microsoft, and Oracle. Which leads to a question that continues to dog the research firms: How much influence do technology vendors have over their work?

Gartner CEO Hall: Trust us

Gartner CEO Hall: Trust us

Photo by Ken Schles
At issue are business practices that beg for closer scrutiny. For example, it's not uncommon for IT research firms to write reports that are funded directly by tech vendors. Money changes hands, and the vendor that commissions a report often reviews it before general distribution. Microsoft's "Get The Facts" marketing campaign has made liberal use of sponsored research to tout the benefits of Windows over Linux. Such reports aren't always clearly marked as having a vendor's backing. A 47-page white paper by Security Innovation, published in November, mentions that it was funded by Microsoft at the bottom of page 6.

Analysts also show up in the marketing programs of the companies they cover. IDC's Bob O'Donnell recently made an appearance in a video produced by thin-client vendor Wyse Technology on the advantages of thin-client computing. IDC also published a report, sponsored by Wyse, that found the software and hardware costs of thin clients to be 40% lower than PCs. Wyse, it turns out, is an IDC client.

And there are hard-to-prove grumblings among small vendors that they have a better chance of being covered by a research firm if they are paying clients. It's called pay-for-play, and it's an issue that the overseers of Gartner's office of the ombudsman do their best to dispel on their Weblog (

InformationWeek went to senior executives of leading IT research and advisory firms to ask how they're addressing questions of objectivity and customer trust. Not surprisingly, all say they're committed to delivering information services of the highest integrity. "We are independent--that is not an issue at all," Gartner CEO Gene Hall says. Maybe, but we also see troubling practices that continue to cast doubt over their best intentions.

Symbiotic Relationships

The experience of Stampede Technologies, a provider of data-acceleration software, shows how the symbiotic relationship between a vendor and an advisory company can create the appearance of a conflict of interest. The company's WebRider software was recently ranked in the "visionary" corner of Gartner's Magic Quadrant for application-delivery products. Stampede wasn't a Gartner customer at the time, but the company did pay the research firm $45,000 as a client in 2004 and even introduced its product at a Gartner conference that year.

Now Stampede is thinking about re-upping with Gartner as it shifts into a new line of business, providing acceleration products for Web applications, says director of marketing Keith Vozel. "For a smaller company such as us, favorable positioning lends us a lot more credibility and gives us a push to get in front of customers we might not have reached before," Vozel says.

There are many companies like Stampede--in search of industry recognition and desperate to get the attention of the big research houses. Proofpoint, a Gartner client since 2003, expects to be included in Gartner's first-ever Magic Quadrant for content monitoring and filtering software, due this quarter. "This matters more than you want it to matter," says Sandra Vaughan, Proofpoint's senior VP of marketing and products.

Failure to get a favorable mention in an analyst report could undermine years of product development. Acceptance, on the other hand, boosts a company's exposure and is essential for buyers drawing up short lists. "Our target market is big companies, so Gartner matters," Vaughan says.

Industry Ties

The IT research and advisory sector has been under pressure for several years as business customers cut back on research spending and turn to cheaper options, resulting in both consolidation and new kinds of services. "There's a lot of free information available," explains Ian Birks, CEO of research firm Ideas International, which two years ago acquired DH Brown Associates.

Gartner, the biggest of IT research companies, last year snapped up Meta Group for $168.3 million, contributing to Gartner's recent growth. Through the first nine months of 2005, its revenue jumped 10% to $699.7 million.

But Gartner's financial statements also reveal a company with ties to the IT industry. The firm invests in hedge funds that hold significant stakes in the companies it covers. One such investment is SI Ventures' SI Venture Fund II. On its Web site, SI Ventures notes a "long-term relationship" with Gartner. SI Ventures helped launch Authentor Systems, which provides network security software. Gartner analysts have been quoted in press releases issued by Authentor supporting the company's approach to security.

Gartner also is partly owned by investment companies that have stakes in tech vendors upon which Gartner is supposed to be casting a neutral eye. Silver Lake Partners, which owns 33% of Gartner, counts Michael Dell, Bill Gates, Larry Ellison, and other tech-industry shakers among its current or former investors. Hedge fund ValueAct Capital owns more than 16% of Gartner and has owned as much as 11% of MSC Software, which Gartner views as a "challenger" in the market for product life-cycle-management software.

While Gartner's investors may not wield direct influence over the firm's product rankings, they do help dictate the company's leadership. Silver Lake and ValueAct, Gartner acknowledges in a Securities and Exchange Commission filing, "may be able to, either individually or together, exercise significant influence on matters requiring stockholder approval." Such matters would include approval of Gartner's senior executives.

Hype Factor

This doesn't mean IT research isn't valuable to many people; thousands of user companies subscribe to research services. Here at InformationWeek, editors regularly talk to analysts to get their perspective on technology trends and forecasts. (InformationWeek also receives millions of dollars in advertising and custom-publishing revenue from technology companies, but ads and "advertorials" are clearly marked and do not involve the editorial staff.)

Gartner's Quadrant for the security information- and event-management market helped Bruce Forman, director of information security at Genesis HealthCare, ensure that he didn't miss any key vendors when he was researching the technology. "It's hard to stay up on this stuff," he says.

UPS spends more than a half-million dollars annually on analyst reports, in-person briefings, and related services. David Barnes, CIO of the package-delivery and logistics company, says he's aware of potential bias but thinks the situation has improved from the days when much of the research was driven by the same hype that powered the dot-com boom. "The companies we've worked with have done a good job of driving that out," he says.

But negative perceptions linger. Some are put off by the fact that researchers typically don't test the products they evaluate. "The firms will talk with vendors and may even talk to people using the technology, but they don't do their own projects," says a technology VP with a large company who requested anonymity.

A visitor to last week weighed in with a different complaint in a blog posting: "The problem I've seen with analyst firms is when analysts forget their primary role as trend watchers and market surveyors and think of themselves as market makers."

Mum's The Word

Research firms and their tech-vendor clients are less than forthcoming about the nature and extent of their relationships. Some IT researchers don't disclose how much of their business comes from tech vendors or which tech vendors are clients. In turn, tech vendors avoid straight answers about how much they spend on IT research. Microsoft, for one, declined to be interviewed for this story.

IBM won't reveal how much it spends on analyst services. John Mihalec, IBM's VP of analyst relations, acknowledges in an E-mail interview that research sponsored by vendors and conducted by analyst firms doesn't always sit well with technology buyers, particularly if they perceive that the research is trying to manipulate purchasing decisions. IBM stipulates that any research it commissions be distributed externally only if it contains "funded by IBM" on the cover page, Mihalec says.

Likewise, Oracle won't say what it spends on IT research. The software developer acknowledges it has contracts with AMR Research to get information on the application market; Yankee Group for the telecom market; Burton Group for security matters; and Forrester, Gartner, and IDC for a range of topics. In addition, Oracle contracts with CCIDnet in China and Bloor Research, Butler Group, and Ovum in Europe. These analysts "have access to competitor information that we wouldn't have or to customers that we don't have," explains Peggy O'Neill, VP of analyst relations.

Oracle occasionally sponsors research, as it did last year when it commissioned a white paper by Crimson Consulting Group to compare the cost of managing application servers. "We'll do a one-off sponsorship for a particular marketing campaign," O'Neill says. But Oracle prefers to encourage analysts to initiate their own independent coverage in areas where research is lacking, a strategy that works in Oracle's favor in several ways: It can promote a new market without paying for the research, and it doesn't have to deal with perceived bias.

IT professionals say they'll sometimes use sponsored research, but they want to be informed that a vendor was involved. "You want to follow up and see if they're leaving anything out," says Lawrence Mathews, IT director with Swinerton, a construction-management company.

Inside The Research Firms

Forrester Research



Yankee Group
245 analysts

1,200 analysts and consultants

775 analysts

80 analysts
Acquired Giga Information Group in 2003; introduced its new "Forrester Wave" report graphic in July 2005

$700 million in revenue for the first nine months of 2005

Privately held subsidiary of IDG known for its market forecasts

Owned by private equity firm Alta Communications; provides IT research, consulting, and events
"Our industry is subject to rumor, it's subject to misinterpretation. Even the perception of being tainted is a problem." --chief strategy officer Brian Kardon

"Analysts don't sell. They are not evaluated on that basis, and don't even know how much revenue we get from each of our clients." --CEO Gene Hall

"Virtually all of our customers are being more careful about what they buy and who they buy it from." --CEO Kirk Campbell

"Even the perception of favoritism would hurt us."
--CEO Emily Green

And there's this twist on the issue of data integrity: Vendors sometimes spin the numbers to their own advantage. Gartner VP Nancy Erskine, in a blog posting, says an ad that Oracle recently ran in The Wall Street Journal "egregiously misrepresents" Gartner's data. The ad claims that 94% of Oracle customers run up-to-date applications compared with only 4% of SAP customers because SAP is "expensive and difficult to upgrade." But Erskine says that data point refers only to products shipped as part of Oracle's E-Business Suite. When all of Oracle's enterprise products are taken into account, a much lower percentage of customers are running the most current versions of software, she says.

Oracle and Gartner have tussled at other times, too. Just last month, Gartner warned CIOs about security issues surrounding Oracle's database software. It goes to show that, symbiotic relationship or not, Gartner and other analyst firms do occasionally crack down on tech vendor clients.

Flawed Analysis

The IT industry's advice dispensers don't always get it right, of course. "I've consistently found that analysts are either stating the relentlessly obvious--'Microsoft has the upper hand in the browser wars'--or making predictions that are flat-out wrong--'Microsoft is going to crush Firefox and sell the spare parts on eBay,'" says Firefox creator Ross Blake.

Three years ago, Gartner research director Richard Stiennon predicted that intrusion-detection systems would no longer be a defense in the security pro's arsenal by 2005. Stiennon, who left Gartner in 2004 to work for anti-spyware developer Webroot Software, contended that organizations would so harden their internal systems that the "burglar alarm" service that intrusion-detection systems provide would no longer be necessary. "Imagine a world where there are no intrusions," he said. (See "Gartner: Intrusion Detection On The Way Out," June 13, 2003.)

Stiennon's report changed the intrusion-detection systems market overnight, says Marcus Ranum, chief security officer for Tenable Network Security and founder of NFR Security, a maker of IDS technology. "That report was so horrifyingly wrong because the thing you're trying to do is detect threats," he says.

Gartner's position on the IDS market caused a stir among potential buyers that went as high as the Pentagon, which in July 2003 called a meeting with IT managers and procurement officials from the Army, Navy, Air Force, Federal Aviation Administration, and departments of Energy, Justice, and Homeland Security to sort out Gartner's analysis. Arbor Networks, Internet Security Systems, NetForensics, NFR Security, and Sourcefire Network Security attended the meeting to present the IDS point of view.

Greg Shipley, CTO at consulting firm Neohapsis, also attended. Gartner's position in the debate was a dangerous one, Shipley now says, because the Pentagon was considering removing IDS from its list of IT spending priorities, though it ultimately didn't. "If there's one group that I want with intrusion detection, it's the Pentagon management team," Shipley says.

Steinnon still stands by his IDS assessment and says "proactive security" is a better approach.

Better Business

Predicting tech trends is notoriously difficult, and no research firm is perfect in its advice. But their motives must be beyond reproach. Gartner VP Larry Perlstein tackled the pay-for-play question on the company's blog. "Aside from the occasional comment on a blog or backroom chatter, we have not had any vendors complain to the office of the ombudsman directly since the office opened in September 2004," Perlstein wrote. "So my message is, if anyone believes that they know of a legitimate issue, please step forward regardless of your client status."

Bias is a nonissue at the company, CEO Hall insists. "We wouldn't have a dollar of revenue from the user community if our objectivity and independence weren't held in high regard," he says. Gartner has policies in place meant to ensure objectivity. The ombudsman office reports to Gartner's general counsel to ensure it's free from pressure from other parts of the company. And Gartner analysts aren't allowed to own stock in the companies they cover.

Execs at other major research firms speak with similar conviction. "We provide fact-based advice," says IDC CEO Kirk Campbell. IDC's research methodology, with its emphasis on hard data, provides a built-in guard against analyst bias or favoritism, he argues. Campbell dismisses any suggestion that vendors have to be paying customers to get fair treatment in IDC reports. "We have an open-door policy," he says.

It sounds convincing, but there's an important bit of information that Campbell refuses to share: He won't disclose how much money IDC takes from tech-vendor clients.

Forrester Research focuses on selling its services to users rather than vendors to ensure that most of its revenue doesn't come from the subjects of its research, says Brian Kardon, chief strategy and marketing officer. That affords the company a lot of freedom. "We routinely slam vendors," he says. Still, about a third of Forrester's revenue comes from tech vendors.

Some research execs concede that their firms must do a better job of educating customers and the public about their policies and procedures, given the influence they have over multimillion-dollar buying decisions. One idea is to develop industrywide standards for business practices. "It's something we should look at," says Yankee Group CEO Emily Green. "Even the perception of favoritism would hurt us." Says Forrester's Kardon, "It would help the whole industry if we had a common set of practices to keep everybody clean."

Aberdeen Group CEO Jamie Bedard last week claimed the high ground. In a progress report on Aberdeen's business, Bedard wrote to customers, "We promised you that ... our research integrity was not for sale." In an interview, Bedard charged that too many research firms base their advice on just a few interviews--what he calls opinion-based research--rather than on detailed surveys of dozens or hundreds of companies. "I think the industry can do a better job of deep research," he says.

On The Radar

The stakes for technology vendors can't be underestimated. Certain vertical industries and large companies look to Gartner for guidance on which products they should include in a technology bake-off, says Vikram Phatak, CTO of intrusion-prevention system maker Lucid Security, a Gartner client since March. This means Lucid must be on Gartner's radar screen when the analyst firm meets with customers who want advice about the IPS market. Phatak expects Lucid to go public or be bought within the next 18 months, so being in the visionary sector of Gartner's IPS Magic Quadrant is crucial. "If you're in the Quadrant, you're a safer bet to potential buyers and VCs than if you're not," he says.

What determines where a vendor lands in Gartner's Quadrant? Gartner says its assessments are based on a vendor's viability, service and support, features and functionality, and technology. Among the criteria that gets factored in: input from customers or prospects who have piloted a product or chosen an alternative; customer experience with scalability and cost of deployment; a vendor's "commitment" in the form of R&D; marketing clarity and sales channel; a vendor's ability to support its vision; and overall corporate viability.

By comparison, Forrester's Wave assesses competitors in a given product category. Each report charts vendor performance on a grid featuring a vertical measurement of a product's strength and horizontal measurement of a company's strategy. Vendors are labeled leaders, strong performers, contenders, or risky bets, and their placement is marked with a dot, the size of which is determined by the vendor's "market presence."

All things considered, the methodology behind Forrester's Wave report is more transparent than Gartner's Quadrant, says Sandi Meyer, senior analyst-relations manager with Trend Micro. That worked to Trend Micro's advantage last month when Forrester listed the vendor as a leader in enterprise anti-spyware. Trend Micro has been a Forrester client since May. The company also subscribes to research from, Gartner, and IDC.

Lessons Learned

Research firm executives are well aware of the questions being raised about their business models, but don't expect changes to be fast or wide-sweeping. The financial stakes are too high--and the incentives for change aren't compelling enough.

Meantime, those vendor-sponsored research reports keep rolling off the presses and into the hands of business-technology buyers. Cyveillance, a provider of online risk monitoring and management software, funded an IDC report published in January titled, "Who's Got Your Virtual Back? Mitigating Online Security Risks." The goal was to provide market education, not to serve as a promo for Cyveillance, VP of marketing Todd Bransford says, so the only place the vendor's name is mentioned is on the cover. "We talked with IDC in order for them to validate what we're doing in the market," Bransford says.

Cyveillance isn't a regular IDC customer. Still, the vendor won't say how much it spent on the IDC report. In advance of publication, Cyveillance had input into the issues that should be covered, and after it was completed, a chance to review it. Cyveillance made no changes to the report's contents before it was released, Bransford says, but it could have requested them, and that's troubling enough. The IDC report preceded the release of Cyveillance's Intelligence Center 3.0 risk-monitoring software by only a few weeks.

Technology vendors often will sign up for analyst firm services when they feel their market is poised for growth. Cyveillance in January signed on with Gartner to help understand how its software fits with the growing demand for IT security and regulatory compliance. "I also wanted to understand how a product we're rolling out should be priced," Bransford says. So IDC helps seed the market, Gartner helps price it, and both get paid for doing so. Is everyone comfortable with that?

Continue to the sidebar:
How Cisco Networks With Analysts

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Email This  | 
Print  | 
More Insights
Copyright © 2019 UBM Electronics, A UBM company, All rights reserved. Privacy Policy | Terms of Service