Strategic CIO // Executive Insights & Innovation
Commentary
8/13/2014
09:30 AM
David F Carr
David F Carr
Commentary
Connect Directly
LinkedIn
Google+
Twitter
RSS
E-Mail
50%
50%

Gartner Magic Quadrant: NetScout Says Secret Is Green

After Gartner analysts rank NetScout only a "challenger," Netscout files lawsuit alleging Gartner's rankings involve pay for play. Let's examine both sides of this street.

"Critics are our friends, they show us our faults." -- Benjamin Franklin

Easy for you to say, Ben. Your bifocals and lightning rods never had to stand up to the judgment of a Gartner Magic Quadrant and the multimillion dollar consequences of being dismissed as a "challenger" or (bigger ouch) a "niche player."

NetScout Systems Inc. certainly doesn't consider Gartner a friend. In a lawsuit filed last week, the seller of network performance boosting technology charged that its classification as a "challenger" in Gartner's Magic Quadrant for Network Performance Monitoring and Diagnostics market was punishment for NetScout's refusal to pay for consulting services over the past five years.

For the uninitiated, the Magic Quadrant is a two-axis graph of a vendor's "completeness of vision" and "ability to execute," where those who get top marks on both are judged "leaders," winning a prized upper-right-hand-corner position (some make it farther to the top and right than others). Those with a lot of vision but not a lot of ability to execute are "visionaries," those with solid ability but not enough vision are "challengers," and those who need more mojo on both dimensions are "niche players."

[Who's the injured party? Read Oracle Fires Back: Oregon Obamacare Exchange.]

NetScout charges that Gartner's consulting business amounts to "pay for play" for placement in research reports. Although Gartner markets itself to enterprise technology leaders as a source of objective and methodical analysis of the strengths and weaknesses of technology vendors, it also sells consulting services to the vendors, coaching them on what they need to do to be more successful in the marketplace. The potential conflict of interest is obvious: one way to be more successful in any IT market is to achieve a top ranking in the Gartner Magic Quadrant.

"In fact, Gartner is not independent, objective or unbiased, and its business model is extortionate by its very nature," says the NetScout filing. "Its substantial success is due to the worst kept secret in the IT industry: Gartner has a 'pay-to-play' business model that by its design rewards Gartner clients who spend substantial sums on its various services by ranking them favorably in its influential Magic Quadrant research reports ('Magic Quadrant reports') and punishes technology companies that choose not to spend substantial sums on Gartner services."

The suit filed in Superior Court in Stamford, Conn., where Gartner is based, calls Gartner's practices a violation of the Connecticut Unfair Trade Practices Act and argues the IT research firm ought to be subject to the same sort of conflict-of-interest rules that govern Wall Street stock analysts.

Gartner says its analysts follow a rigorous code of conduct and their work is subject to review by its Office of the Ombudsman, which reviews vendor complaints. In a blog post maintaining that client status is irrelevant to positioning in its reports, Gartner states that if vendors are buying services to improve their position in its Magic Quadrant or Marketscope reports, "they quickly find that becoming a client doesn't guarantee either of these." Any vendor can brief Gartner on its products, according to the post, and the only additional thing paying vendor clients get is Gartner's feedback.

The way this is supposed to work is that IT buyer clients get the benefit of Gartner's analysis of what the vendors can deliver, while IT vendor clients get the benefit of what Gartner has learned about the buyer requirements and practical challenges, and the analysis in both cases is supposed to be objective. However, vendors who are paying customers can undeniably get more time with the analysts, meaning the analysts will know them and their products better and perhaps think better of them (when it comes time to do that "objective" analysis) than they would otherwise.

The lawsuit quotes an unnamed Gartner analyst as telling NetScout's president and CEO, "NetScout is not going anywhere because it does not spend enough on marketing." The filing adds, "NetScout's CEO reasonably understood this statement to mean that NetScout should spend more money on Gartner's services."

Even if NetScout could document that this exchange took place, would a judge be willing to read between the lines the same way?

The pay-to-play charge is not a new criticism for Gartner. A couple of years ago, ZL Technologies sued and lost over a similar complaint, saying that it had been "defamed" by a harsh evaluation from Gartner. Yet in the most recent Magic Quadrant for Enterprise Information Archiving, Gartner ranked ZL Technologies as a leader -- and ZL issued a press release bragging of its favorable ranking under what it previously argued was a corrupt and unreliable methodology.

Gartner prevailed in the ZL Technologies dispute partly by arguing that its rankings, although advertised as objective, are ultimately opinion and cannot be judged true or untrue like objective facts. NetScout argues it deserved a higher ranking, but it also claims the Magic Quadrant report contained factual errors that it objected to after reviewing a draft of the report -- but Gartner refused to correct them. For example, Gartner described NetScout as offering "only a hardware-based deployment model" that "limits [its] ability to address growing software and SaaS solution demand." NetScout said Gartner refused to acknowledge its efforts to address those market changes and unfairly characterized it as struggling to modernize aging technology.

When NetScout asked to be removed from the report entirely, Gartner refused, saying the vendor was enough of a major player that the report would be incomplete and lack credibility if NetScout was omitted.

The creators of IT products have long grumbled about having to kowtow to powerful market analyst firms and questioned whether those, like Gartner, who claim to act as an advocate for IT buyers are doing so according to fair criteria. One of the many sources the lawsuit quotes to substantiate those doubts is a 2006 InformationWeek analysis on the Credibility of Analysts.

Passages from our report quoted in the lawsuit include this, from the introduction:

[T]hey... rake in millions providing services to the very same companies they monitor, heavyweights like Cisco, IBM, Microsoft, and Oracle. Which leads to a question that continues to dog the research firms: How much influence do technology vendors have over their work?

At issue are business practices that beg for closer scrutiny.

The lawsuit cites one other excerpt from the same InformationWeek report:

Proofpoint, a Gartner client since 2003, expects to be included in Gartner's first ever Magic Quadrant for content monitoring and filtering software . . . . "This matters more than you want it to matter," says Sandra Vaughan, Proofpoint's senior VP of marketing and products.

Failure to get a favorable mention in an analyst report could undermine years of product development. Acceptance, on the other hand, boosts a company's exposure and is essential for buyers drawing up short lists. "Our target market is big companies, so Gartner matters," Vaughan says.

Even more entertainingly, the report quotes company founder Gideon Gartner from remarks at the Computer History Museum agreeing with an audience member who called the Magic Quadrant one of the most "reviled" market measures in the industry. "The reason why people revile the Magic Quadrant is because it is misused," Gartner said. "As a guideline for a bunch of amateurs it's one thing. But when all your clients live or die on the basis of whether Gartner Group puts you in the upper right hand corner in the -- or wherever -- that's really bad news. And when there's potential tainting of the objectivity of research because you have very large contracts with your vendors, with your customers, people will always come and complain." He said the analytic measure was "overused, misused, and abused, terribly." (Note that Mr. Gartner is no longer part of Gartner Inc. and subsequently founded competing companies.)

Richard Stiennon, a former Gartner Research VP and the author of UP and to the RIGHT: Strategy and Tactics of Analyst Influence, a book of advice for technology vendors, blogged about NetScout's move as a "great blunder" and one that CIOs in its target market are unlikely to read as anything but whining. He finds NetScout's reaction to its Magic Quadrant ranking "overblown," particularly given that the analysts also said some very positive things about the NetScout products and company. Stiennon said

Next Page

David F. Carr oversees InformationWeek's coverage of government and healthcare IT. He previously led coverage of social business and education technologies and continues to contribute in those areas. He is the editor of Social Collaboration for Dummies (Wiley, Oct. 2013) and ... View Full Bio
Previous
1 of 2
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
<<   <   Page 2 / 2
Laurianne
50%
50%
Laurianne,
User Rank: Author
8/13/2014 | 2:43:39 PM
Magic Quadrant
Magic Quadrant has been controversial for years of course. But as Dave points out, what it hasn't been is ignored. In the age of Twitter, mock Magic Quadrant graphics have become a humor form in themselves.
David F. Carr
50%
50%
David F. Carr,
User Rank: Author
8/13/2014 | 10:23:53 AM
Re: Opening Doors
Even Gartner says limiting your selection to only the firms in the leaders quadrant is unwise, since those judgments are necessarily general. The best firm for your purposes might actually be a "niche vendor" that happens to target the niche in the market you fit into. For example, the best large enterprise solution often is not the best for a startup or midsize company.
Alison_Diana
100%
0%
Alison_Diana,
User Rank: Author
8/13/2014 | 10:11:57 AM
Opening Doors
Recently I interviewed the CIO of a healthcare organization that cited Gartner's Magic Quadrant as the major way it selected vendors to review. Rather than wade through tens of vendors in that space, the CIO found this tool to be a useful way to reduce the number of candidates -- and it ultimately awarded the contract to a company in the Quadrant, of course. OTOH, another IT exec used Gartner's analysis for the same reason but also included a non-participating vendor (a smaller company) -- and that smaller firm won the deal. As a reporter, news that a company has been named to one quadrant or another is something i don't even consider.
<<   <   Page 2 / 2
The Business of Going Digital
The Business of Going Digital
Digital business isn't about changing code; it's about changing what legacy sales, distribution, customer service, and product groups do in the new digital age. It's about bringing big data analytics, mobile, social, marketing automation, cloud computing, and the app economy together to launch new products and services. We're seeing new titles in this digital revolution, new responsibilities, new business models, and major shifts in technology spending.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Dec. 9, 2014
Apps will make or break the tablet as a work device, but don't shortchange critical factors related to hardware, security, peripherals, and integration.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of December 14, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.