InformationWeek
Defining The Business Value Of Technology
November 17, 1997
Analyzing The Analysts
By Bob Violino and Rich Levin
|
View survey results, "
How They Stack Up
" as a PDF file.
View the chart, " What The Analysts Offer " as a PDF file.
SEE RELATED STORIES: The Crusader : Dale Kutnick, Meta Group
The Up & Comer
: Gene DeRose,
The Patriarch
: Gideon Gartner,
The Golden Gut
: George Colony,
To view a PDF file,
download t he Adobe Acrobat Reader . |
Still, the cornerstone of the services offered by these companies is to help IS leaders choose the products and technologies that represent the best bets in their environment. Some analysts take a quantitative approach to market research, while others work more subjectively. Either way, IS managers apparently put a lot of faith in the analysts' ability to predict the success of products and technologies: When asked which analyst reports their company buys, more than 90% of the executives in the IW survey said they bought product and technology forecasts.
Many companies select a combination of analyst firms to best suit their needs. Cigna, for instance, relies on Gartner Group for strategic IT planning and design, but uses Forrester to track broad business and market trends. Similarly, Kodak uses Meta services for basic IT strategy planning, Gartner for "spot" applications such as sales-force automation, and Giga for Web-based services.
"Our recommendation to users is that it's always wise to work with multiple analysts," says Digital's Scull. "It's the same advice you'd give someone about medical care-look at second opinions."
Even the analyst firms agree that there are profound differences among them. Gartner Group, for example, emphasizes a rigid research methodology, enforced by a custom-developed software program and overseen by an army of analysts. Gartner's 550 analysts worldwide cover 66 research services across 400 core IT topics, according to Bill Clifford, president of Gartner Group Research. Its client list includes some 8,300 companies.
Meta Group, an analyst firm in Stamford, Conn., puts a premium on personal interaction and consultation with clients. At the same time, it de-emphasizes published report s. "We've structured our service to be able to talk with our customers," says Joe Gottlieb, Meta's executive VP of marketing. "This allows us to provide clients with ideas, trends, and guidance within the context of their business scenario."
Forrester Research, in Cambridge, Mass., has some 45 analysts and 25 researchers, and it focuses on primary research among early adopters of technology. "If you want to understand the adoption of new technologies, you have to understand the user's viewpoint," says Bill Bluestein, VP of new-media research. "Our clients know that even though we have outrageous opinions at times, or contrarian insight, it's based on some serious research. It's not just us sitting here getting briefed by slick technology-marketing people."
Giga Information Group, in Norwell, Mass., was created by Gartner Group founder Gideon Gartner. It's perhaps best known for GigaWeb, the first online service to provide subscribers with instant access to the entire body of the organization's research.
The Yankee Group, an $8 million firm, specializes in communications services. "We still regard communications as the dominant technology," says Anderson, who is also the Yankee Group's founder. "Gartner does much better residual analysis on mainframes than we ever did. Mainframe software, Meta and Gartner are pretty good at that. But for datacom networks and enterprise applications, clients say, `I'll use the Yankee Group for that.'" (The Yankee Group was acquired by Primark Corp. of Waltham, Mass., last year.)
International Data Corp. in Framingham, Mass., employs more than 300 market analysts and produces 2,000 IT market surveys a year. The company, with offices in 40 countries, emphasizes international coverage.
Smaller market research organizations such as Aberdeen, Jupiter, and Zona focus on specific technologies. For example, Hurwitz Group, a 13-analyst firm in Newton, Mass., specializes in software. "I started this firm based on the concept that what was most important, in terms of analysis, was an integrated view of software," says Judith Hurwitz, the firm's president (and an IW columnist).
Smaller analyst firms can also become acquisition bait for the giants. In late October, Gartner Group invested $8 million in Jupiter, gaining a 32% stake in the New York firm and a seat on its board. Jupiter specializes in the consumer-Internet market and new-media research.
With all these firms vying for business-not to mention the many other firms that exist-how can IS managers differentiate one analyst from another? Executives in the IW survey recommend looking first to the quality of the firm's advice. Other important factors include reliability, overall reputation, and accurate product forecasts, they say.
With the growth of the IT analyst business from a handful of firms in the 1980s to dozens of large and small companies today, the choices have become more complex than e ver. But based on the IW survey and interviews with executives, Gartner Group is by far the dominant analyst firm. Nearly 90% of the executives surveyed by IW say they have purchased research reports from the Stamford firm or its Dataquest subsidiary in the past 12 months.
When the executives were asked which analyst firm they'd pick if they could use only one, Gartner was the choice of more than 71%. Gartner also scored high when IT executives were asked to rate their satisfaction with the analyst firms in a variety of categories: It ranked first or second in all 10 categories. Only Jupiter Communications scored higher, with a far smaller but loyal following.
Gartner Group's influence on the marketplace has been nothing short of profound. Its estimates on year 2000 conversion costs have been widely quoted and are the basis for cost estimates at many organizations. Similarly, G
artner's estimates for TCO-the total cost of PC ownership-formed the basis of industrywide debates on the merits of network computers, even though not everyone agreed with Gartner's findings. This week, Gartner will demonstrate a new TCO methodology, based on research it conducted with 10 IT vendors, to help users determine their enterprisewide IT costs of ownership, including PC, network, and server costs.
But with Gartner Group's reach and clout comes extra scrutiny. And when some IT users and vendors look closely at Gartner Group, they don't always like what they see.
One concern centers on Gartner Group's so-called Magic Quadrants. These are four-part grids that Gartner analysts use to graphically depict IT vendors' strengths and weaknesses, and they've become one of the industry's most powerful tools. Gartner analysts can help make or break products, technologies, or services, depending on a vendor's location on the quadrant. The upper right-hand quarter is the sweet spot; the bottom-left, the sour grapes. The only thing worse is not appearing in any quadrant at all.
Gartner's Clifford says the Magic Quadrants are a way the firm "lays out the landscape of a sector of the IT industry." But some vendors charge that a spot on a Magic Quadrant slide requires little more than a competent technology and a Gartner sales call. Others claim that the firm conducts unfair analysis based on whether a vendor is one of its clients. "We did a little preview at Gartner, and they slammed us," says a marketing director at one systems software firm who requested anonymity. "So we hired them to do a technology assessment. We paid them, basically, to take a closer look at us."
After the technology assessment was complete, the marketing executive continues, Gartner officials told him the technology was of high quality. "Until we paid them to do [a technology assessment], it was tough to get noticed," he says. It doesn't matter how technology-savvy you are. The point is, I've got to buy consulting services fro m Gartner. That's the price I have to pay to get Gartner's blessing."
Dave Passmore, a former Gartner Group analyst and now president of Decisys Inc., a consulting firm in Sterling, Va., won't go that far, but he says vendors sign up with Gartner for two reasons: "One, because of this perception that if they don't, Gartner won't write good things about them. Two, they're dying to know what Gartner writes about them."
Gartner officials strongly deny that the firm's influence is for sale. "We've got very stringent policies against that," says Clifford. "There is a complete separation of church and state here in terms of our relationship with a vendor as a client vs. our relationship with a vendor who is a subject of our research. We will not-nor have we ever-accepted fees for publication."
Executives in the IW survey give Gartner high ratings for credibility. Some also cite Gartner's high standards and lack of bias. "Gartner is very objective," says Trudell of U.S. Steel. "They don't hesitate to gi ve both the good and the bad about vendors or technology."
But even Trudell was disappointed when Gartner's most recent annual IT symposium didn't include participation by Computer Associates or Oracle. He attributes the omissions to what he calls "bad blood" between Gartner and the two vendors.
While Oracle officials failed to return telephone calls, Marc Sokol, senior VP of advanced technology at Computer Associates in Islandia, N.Y., claims Gartner has a de facto bias against his company. "We don't get a fair shake from them," he says. "I believe that any analyst who will write anything positive in print about CA will be stopped by Gartner's editorial process. We've had meetings together, our CEOs have met, but we can't even get as far as a fair, unbiased relationship." Sokol adds that "other analyst firms have blasted us in the past, but we still have a good relationship with them."
But Myron Kerstetter, VP of research management at Gartner Group, denies that the firm has any bias against CA. "It's difficult to understand why we'd have any bias," he says. "There's no benefit to us." Instead, Kerstetter suggests, "We're trying to provide council and suggestions and the most accurate and useful information to our client base, sometimes in a way that's difficult for the vendors to stomach."
Indeed, several other vendors find the analysts are free of bias. "If anything, most of them go overboard to avoid that," says Ken Conway, manager of analyst relations at Compaq Computer. "They're pretty much out of business if they choose a favorite." Gail Daniels, VP of worldwide marketing at IBM's Tivoli unit, agrees: "At the end of the day, we wouldn't do business with them if we didn't think they were fair."
Room To Grow
IS executives also say there's room for improvement in the way virtually all the analyst firms gather, analyze, and present information, particularly in light of their growing client bases and complexity of the IT business. "I see two major challenges faced by all of thes
e firms," says Spangenberg of Prudential. "First, they have to make their knowledge of specific value to the companies they serve. Second, with the information explosion going on, they have to not only gather data, but also make sense of it and communicate that to customers. They have a hard task ahead of them to stay ahead of the curve. They're struggling to keep pace."
Others would like to see the analysts scored on the accuracy of their predictions, the way Wall Street stock pickers are, and to have this information made publicly available. CA's Sokol agrees there needs to be more accountability. "Industry analysts make incorrect predictions constantly," he says. "They said OS/2 would be the dominant platform; they said everything would run SAA."
Still other IS managers want to take a much closer look at consulting firms' performance in general, not just their forecasting. P
ete Schmidt, general director of strategic planning in General Motors Corp.'s IS and services organization, is reviewing his subscriptions with Gartner and Forrester. He wonders if he should be "putting together a matrix for deciding which research firms we should be using to figure out if we are getting value for what we invest in them."
Beyond the issue of whether the research services provide value, Schmidt wants to know: If there are reports, who looks at them? If phone time with analysts is included in the fees, is anyone using that time? Have the right people subscribed? Are we utilizing what we're paying for?
Some also say the analysts should be required to reveal if and when they have financial interest in the companies they cover. "If there's any kind of influence like that, it ought to be known right up front, so you can temper your opinion," says Trudell of U.S. Steel.
In fact, some analyst firms have rules against questionable behavior. Giga, for example, has an ethics policy that pr ecludes "doing anything that could be a conflict of interest or an exploitation of insider information," says David Gilmour, the firm's chief research officer. Similarly, Meta forbids its analysts from providing quotes for use in vendor marketing materials or press releases.
One key to using analysts is to clearly define the role analysts play. Expecting them to do everything is a recipe for disaster.
Kodak's decision to move its desktops to Windows NT, for example, was just that-Kodak's decision. While the analysts provided perspective and guidance, the ultimate move was Kodak's. "The analysts give us the advice," says Royston. "But we have to figure out where we're going."
--with additional reporting by Caryn Gillooly , Marianne Kolbasuk McGee , and Bruce Caldwell , and with research assistance from Gerald Lazar
