Strategic CIO // Executive Insights & Innovation
Commentary
8/13/2014
09:30 AM
David F Carr
David F Carr
Commentary
Connect Directly
Twitter
LinkedIn
Google+
RSS
E-Mail
50%
50%

Gartner Magic Quadrant: NetScout Says Secret Is Green

After Gartner analysts rank NetScout only a "challenger," Netscout files lawsuit alleging Gartner's rankings involve pay for play. Let's examine both sides of this street.

"Critics are our friends, they show us our faults." -- Benjamin Franklin

Easy for you to say, Ben. Your bifocals and lightning rods never had to stand up to the judgment of a Gartner Magic Quadrant and the multimillion dollar consequences of being dismissed as a "challenger" or (bigger ouch) a "niche player."

NetScout Systems Inc. certainly doesn't consider Gartner a friend. In a lawsuit filed last week, the seller of network performance boosting technology charged that its classification as a "challenger" in Gartner's Magic Quadrant for Network Performance Monitoring and Diagnostics market was punishment for NetScout's refusal to pay for consulting services over the past five years.

For the uninitiated, the Magic Quadrant is a two-axis graph of a vendor's "completeness of vision" and "ability to execute," where those who get top marks on both are judged "leaders," winning a prized upper-right-hand-corner position (some make it farther to the top and right than others). Those with a lot of vision but not a lot of ability to execute are "visionaries," those with solid ability but not enough vision are "challengers," and those who need more mojo on both dimensions are "niche players."

[Who's the injured party? Read Oracle Fires Back: Oregon Obamacare Exchange.]

NetScout charges that Gartner's consulting business amounts to "pay for play" for placement in research reports. Although Gartner markets itself to enterprise technology leaders as a source of objective and methodical analysis of the strengths and weaknesses of technology vendors, it also sells consulting services to the vendors, coaching them on what they need to do to be more successful in the marketplace. The potential conflict of interest is obvious: one way to be more successful in any IT market is to achieve a top ranking in the Gartner Magic Quadrant.

"In fact, Gartner is not independent, objective or unbiased, and its business model is extortionate by its very nature," says the NetScout filing. "Its substantial success is due to the worst kept secret in the IT industry: Gartner has a 'pay-to-play' business model that by its design rewards Gartner clients who spend substantial sums on its various services by ranking them favorably in its influential Magic Quadrant research reports ('Magic Quadrant reports') and punishes technology companies that choose not to spend substantial sums on Gartner services."

The suit filed in Superior Court in Stamford, Conn., where Gartner is based, calls Gartner's practices a violation of the Connecticut Unfair Trade Practices Act and argues the IT research firm ought to be subject to the same sort of conflict-of-interest rules that govern Wall Street stock analysts.

Gartner says its analysts follow a rigorous code of conduct and their work is subject to review by its Office of the Ombudsman, which reviews vendor complaints. In a blog post maintaining that client status is irrelevant to positioning in its reports, Gartner states that if vendors are buying services to improve their position in its Magic Quadrant or Marketscope reports, "they quickly find that becoming a client doesn't guarantee either of these." Any vendor can brief Gartner on its products, according to the post, and the only additional thing paying vendor clients get is Gartner's feedback.

The way this is supposed to work is that IT buyer clients get the benefit of Gartner's analysis of what the vendors can deliver, while IT vendor clients get the benefit of what Gartner has learned about the buyer requirements and practical challenges, and the analysis in both cases is supposed to be objective. However, vendors who are paying customers can undeniably get more time with the analysts, meaning the analysts will know them and their products better and perhaps think better of them (when it comes time to do that "objective" analysis) than they would otherwise.

The lawsuit quotes an unnamed Gartner analyst as telling NetScout's president and CEO, "NetScout is not going anywhere because it does not spend enough on marketing." The filing adds, "NetScout's CEO reasonably understood this statement to mean that NetScout should spend more money on Gartner's services."

Even if NetScout could document that this exchange took place, would a judge be willing to read between the lines the same way?

The pay-to-play charge is not a new criticism for Gartner. A couple of years ago, ZL Technologies sued and lost over a similar complaint, saying that it had been "defamed" by a harsh evaluation from Gartner. Yet in the most recent Magic Quadrant for Enterprise Information Archiving, Gartner ranked ZL Technologies as a leader -- and ZL issued a press release bragging of its favorable ranking under what it previously argued was a corrupt and unreliable methodology.

Gartner prevailed in the ZL Technologies dispute partly by arguing that its rankings, although advertised as objective, are ultimately opinion and cannot be judged true or untrue like objective facts. NetScout argues it deserved a higher ranking, but it also claims the Magic Quadrant report contained factual errors that it objected to after reviewing a draft of the report -- but Gartner refused to correct them. For example, Gartner described NetScout as offering "only a hardware-based deployment model" that "limits [its] ability to address growing software and SaaS solution demand." NetScout said Gartner refused to acknowledge its efforts to address those market changes and unfairly characterized it as struggling to modernize aging technology.

When NetScout asked to be removed from the report entirely, Gartner refused, saying the vendor was enough of a major player that the report would be incomplete and lack credibility if NetScout was omitted.

The creators of IT products have long grumbled about having to kowtow to powerful market analyst firms and questioned whether those, like Gartner, who claim to act as an advocate for IT buyers are doing so according to fair criteria. One of the many sources the lawsuit quotes to substantiate those doubts is a 2006 InformationWeek analysis on the Credibility of Analysts.

Passages from our report quoted in the lawsuit include this, from the introduction:

[T]hey... rake in millions providing services to the very same companies they monitor, heavyweights like Cisco, IBM, Microsoft, and Oracle. Which leads to a question that continues to dog the research firms: How much influence do technology vendors have over their work?

At issue are business practices that beg for closer scrutiny.

The lawsuit cites one other excerpt from the same InformationWeek report:

Proofpoint, a Gartner client since 2003, expects to be included in Gartner's first ever Magic Quadrant for content monitoring and filtering software . . . . "This matters more than you want it to matter," says Sandra Vaughan, Proofpoint's senior VP of marketing and products.

Failure to get a favorable mention in an analyst report could undermine years of product development. Acceptance, on the other hand, boosts a company's exposure and is essential for buyers drawing up short lists. "Our target market is big companies, so Gartner matters," Vaughan says.

Even more entertainingly, the report quotes company founder Gideon Gartner from remarks at the Computer History Museum agreeing with an audience member who called the Magic Quadrant one of the most "reviled" market measures in the industry. "The reason why people revile the Magic Quadrant is because it is misused," Gartner said. "As a guideline for a bunch of amateurs it's one thing. But when all your clients live or die on the basis of whether Gartner Group puts you in the upper right hand corner in the -- or wherever -- that's really bad news. And when there's potential tainting of the objectivity of research because you have very large contracts with your vendors, with your customers, people will always come and complain." He said the analytic measure was "overused, misused, and abused, terribly." (Note that Mr. Gartner is no longer part of Gartner Inc. and subsequently founded competing companies.)

Richard Stiennon, a former Gartner Research VP and the author of UP and to the RIGHT: Strategy and Tactics of Analyst Influence, a book of advice for technology vendors, blogged about NetScout's move as a "great blunder" and one that CIOs in its target market are unlikely to read as anything but whining. He finds NetScout's reaction to its Magic Quadrant ranking "overblown," particularly given that the analysts also said some very positive things about the NetScout products and company. Stiennon said

Next Page

David F. Carr oversees InformationWeek's coverage of government and healthcare IT. He previously led coverage of social business and education technologies and continues to contribute in those areas. He is the editor of Social Collaboration for Dummies (Wiley, Oct. 2013) and ... View Full Bio
Previous
1 of 2
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 2   >   >>
Laurianne
50%
50%
Laurianne,
User Rank: Author
8/14/2014 | 2:06:32 PM
Re: Cloud Magic Quadrant reflects changing fortunes, not who has a fortune
The cloud service providers certainly debate their placements with vigor -- and often snarky humor.
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
8/14/2014 | 1:37:00 PM
Cloud Magic Quadrant reflects changing fortunes, not who has a fortune
I wouldn't say this for all Magic Quadrants because I don't know enough, but I think the Cloud MQ is done with rigor, changing from year to year as the fortunes of different service providers rise and fall. If I were looking for the influcence of money in analyst activities, I wouldn't question the MQ. I'd look to the analyst comments often cited in company product announcements. Gartner's business status is tied up in high profile MQ and it would lose value if it weren't related to merit, with so many jealous and zealous eyes watching.
Lorna Garey
50%
50%
Lorna Garey,
User Rank: Author
8/14/2014 | 11:55:27 AM
Re: or Gartner consulting
Exactly, Rob. I edited plenty of those reviews, and let's be clear, the model was a good one that the vendors whining about paying Gartner didn't support with their advertising dollars or, often, by taking part. I often hear IT pros on both sides of the vendor/buyer aisle wax nostalgic for the days when publications like NWC could support labs that produced unbiased reviews by experienced technology journalists, not paid analysts. 

At this point, if vendors feel they are under the thumb of analyst firms, they ought to look in the mirror before affixing blame.
irish1
50%
50%
irish1,
User Rank: Apprentice
8/14/2014 | 10:19:43 AM
Pay for Play? Really?
All of us in the industry understand the iinfluence Gartner has with buyers. It is a high stakes process for sure, but I frankly always find this "pay for coverage" argument annoying. I have been in multiple start-ups and mid-size software companies that have been fairly evaluated by Gartner, Forrester, and others - and I can guarantee you that in none of these cases we had the funding to spend a "material" amount of money with Garnter. I have always found that independent of spending level, if you have a good product, customers, value proposition, and process to work with Gartner, you will be fairly recognized in their research. The spend level too often is used as an excuse IMO.
RobPreston
50%
50%
RobPreston,
User Rank: Author
8/14/2014 | 9:35:03 AM
Re: or Gartner consulting
When I was an editor at Network Computing, back when NWC did comparative product reviews, we gave lots of vendors/products poor grades after doing rigorous product testing. I was always impressed with those poorly rated vendors who used it as a learning opportunity: They came back to our product testers and asked what could they do better and how. We didn't charge them for that advice, of course. 
David F. Carr
50%
50%
David F. Carr,
User Rank: Author
8/14/2014 | 9:17:41 AM
Re: Gartner Magic Quadrant: NetScout Says Secret Is Green
Re: "don't get involved in the MQ process in the first place," NetScout actually discovered that it couldn't just opt out of being included in the report because Gartner considered it too important a player in the market sector. Maybe smaller firms could simply avoid inclusion by not volunteering themselves.
brudowitz070
50%
50%
brudowitz070,
User Rank: Apprentice
8/14/2014 | 9:12:42 AM
Gartner Magic Quadrant: NetScout Says Secret Is Green
The MQs are simply a tool for customers to use to get some perspective on prospective vendors or products. Our company is in two MQs and we land in the Niche space which is exactly where we belong. We see the Strengths and Cautions as very approrpriate and we understand we are not ever going to be in the upper right because we compete against major global firms in our spaces who have the wherewithal to be leaders.

We do not spend consulting dollars with Gartner but have used their research for a number of years. I have used it in previous positions. I find it helpful in narrowing down my searches and useful in making decisions about the vendors we choose to interview. However, I would never solely base any purchase decision on anyone elses research. This seems to be another frivolous lawsuit brought by someone who simply did not like where they were placed.

If you're unsure about what will happen, don't get involved in the MQ process in the first place. Sour grapes if you ask me.
ron.cleaver
100%
0%
ron.cleaver,
User Rank: Apprentice
8/14/2014 | 8:43:32 AM
Re: Opening Doors
I have read many of the Magic Quadrant reports.  While I find some of them useful for identifying potential vendors to evaluate, it many cases I happen to have intimate knowledge of the vendor because my team evauated them in the past.  I find the Gartner reports to be very superficial.  I also find their approach questionable at best, as if the key to success is marketing.  That's like saying whoever tells the biggest lies wins.  I don't accept any marketing claims at face value,  Hence the need for a detailed evauation, as well as a trial evaluation, no matter which vendor your choose,
thunter296
100%
0%
thunter296,
User Rank: Apprentice
8/14/2014 | 7:21:15 AM
or Gartner consulting
When I was CEO at ConnectYourCare, we scored in the top right hand quadrant twice if I recall correctly. We didn't spend a penny for Gartner consulting...but we did listen to their observations and addressed these both in our product offerring and in the way we marketed our cloud-based SaaS product.

In a former life at The Hunter Group, we went through a rigorious evaluation to earn top ranking. In fact, Gartner hired us for their enterprise software implementation.

I've always found their analysts fair and really knowledgeable.

Terry Hunter

 

 

 
Brian.Dean
100%
0%
Brian.Dean,
User Rank: Ninja
8/14/2014 | 5:35:06 AM
Re: Opening Doors
Excellent article about a Catch 22 situation in the IT space, Gartner needs to sell some services in-order to generate revenue to pay their analysts. The greater a partnership, the greater Gartner knows about the vendor and the vendor would gain valuable feedback from a firm that understands a number of players in the market -- helping to improve the vendor's business process -- increasing their revenue. In turn, the vendor would be ranked with 5 stars rather than, 4 stars.

And, all this is in the IT space, meaning, in two years the market would have already shifted towards a new trend.

It would be interesting to see how the law deals with this situation and the timeframe that is required. I feel NetScout has the capital at hand to improve their business. Otherwise, they would have not indulged in a lawsuit. However, I feel, it would have been better if they had taken up the services of a third-party analysis firm and tested it for a year to see if any positive movement was taking place for their firm in Gartner Magic Quadrant.

 
Page 1 / 2   >   >>
The Business of Going Digital
The Business of Going Digital
Digital business isn't about changing code; it's about changing what legacy sales, distribution, customer service, and product groups do in the new digital age. It's about bringing big data analytics, mobile, social, marketing automation, cloud computing, and the app economy together to launch new products and services. We're seeing new titles in this digital revolution, new responsibilities, new business models, and major shifts in technology spending.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Must Reads Oct. 21, 2014
InformationWeek's new Must Reads is a compendium of our best recent coverage of digital strategy. Learn why you should learn to embrace DevOps, how to avoid roadblocks for digital projects, what the five steps to API management are, and more.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
A roundup of the top stories and trends on InformationWeek.com
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.