The Reality Of Research

International Truck and Engine Corp. is more selective in its choice of IT analyst firms these days. Last year, as the economy roared along with the force of an 18-wheel tractor-trailer, the truck manufacturer and Ford Motor Co. engine supplier could afford to buy a wide range of analyst reports and services from four firms. But a $30 million loss in the first nine months of this year, compared with a $264 million profit in the same period the previous year, has forced the company to tighten its belt. So Art Data, VP of IT at the Warrenville, Ill., company, worked with his staff to figure out which analyst firms International had confidence in and which hadn't lived up to expectations. The team decided that one firm (Data won't say which) didn't provide the relevance and depth of understanding International required, and it was dumped. Those retained--AMR Research, Gartner, and Meta Group--are looking at less money because International decided to significantly cut its annual $350,000 IT research budget. Data signed new contracts with AMR and Meta Group valued at about $200,000, and he's using Gartner for special projects with fees negotiated up-front on each venture.

A bit of irony in the situation wasn't lost on Data. Earlier this year, one analyst firm was floating a 20% increase in its fees--after "the last research we'd purchased from them focused on how we could reduce costs," Data says. His answer: "No way." The firm withdrew its price increase in order to keep International as a customer.

It's little wonder that analyst firms are open to compromise these days. Despite growth projections by the largest firms, the sector in general is feeling the pinch of the strained economy right along with those who buy analysts' services--maybe even more so. In a down economy, companies concentrate on must-haves, and research services aren't necessarily that. With companies watching their budgets closely, it's getting harder for IT executives to justify buying research reports and services--in a new InformationWeek Research survey of 175 business technology professionals, more than half say the economy has made it harder to make a case for these purchases.

They're particularly troubled when they look back at the exuberance with which some analyst firms, at the height of the dot-com frenzy, recommended they embrace strategies, technologies, and vendors that have since fizzled. A significant number of managers also are concluding that analyst firms they've used have failed to deliver results in key areas, such as providing insights about customers or anticipating business shifts, according to our survey. Equally problematic during a time when executives are demanding a clear-cut return on investment for every IT expenditure, a quarter of respondents surveyed say their ROI from IT research is low or nonexistent.

It's not all bad news for analyst firms, though. The survey finds that a majority of those polled believe their primary IT research firm's analysis is accurate and its advice is clear and unbiased. More than a quarter of respondents say they will purchase more research reports and services than they did last year, and more than half say they'll purchase the same, which may help justify the expectations of top firms that spending on their reports and services will grow by more than 10% this year.

Still, potential clients are taking a closer look. "There's a lot of research out there that doesn't seem to be based in reality," says Terry Edney, senior VP of strategy and marketing for Siemens Business Services, whose parent company, Siemens AG, is deploying an $800 million supply-chain system worldwide. He says analysts' recent enthusiasm for business-to-business strategies has been just as troublesome as their fervor for business-to-consumer initiatives was last year--particularly analysts who fail to consider how much time it can take for B-to-B technology to deliver on its promise. "We can't just look at what's new in technology. We need to know its business impact and how quickly it can be implemented in an environment where there's already an existing IT infrastructure," Edney says.

"There's a trough of disillusionment around IT research," admits Michael Fleisher, CEO of Gartner, one of the most well-regarded analyst firms. Fleisher blames the credibility gap on the failure of many analyst firms to spot the dot-com bust in advance, the hype surrounding pure-play business-to-consumer E-commerce, and what he calls an "irrational enthusiasm" for everything B-to-B. That, as much as a tough economy, is prompting IT research buyers to question the research they receive, he says.

Some clients feel they've been burned by too much enthusiasm. While 84% of survey respondents gave analyst firms high marks regarding E-business strategy expertise, 16% say they've done a poor or unsatisfactory job understanding the topic. "Where did analysts help us by encouraging a focus on B-to-C?" asks Scott Irwin, a VP with Aktion Associates Inc., a Maumee, Ohio, IT consulting firm. He says he can't estimate the amount of money his firm may have lost by focusing on that strategy--at the recommendation of several analyst firms. Dave Viertels, a senior account manager at 24/7 Broadband and Professional Services, a systems integrator and custom software development subsidiary of 24/7 Media Inc., concurs that analysts have a tendency to embrace technologies too quickly when they become popular. "You can't blame them," he says. "You just have to be aware of it and look for research that balances against all the hype."

Though more than half of respondents to the InformationWeek Research survey say they receive only moderate return on investment for their IT research dollars, just 5% of respondents say cost is a negative factor when dealing with analyst firms. That may be because respondents receive almost half their research reports and services free from the firms and sources such as vendors and colleagues.

A breakdown of spending plans among survey respondents shows that 55% will spend more than $40,000 this year on IT research, 29% up to $150,000, 13% up to $1 million, and 13% more than $1 million. Whether those spending plans hold true, though, may depend on the economy doing the same.

"There's no question we're all playing in a tough economy," says Fleisher, who predicts a continued shakeout among IT research firms in coming months. "Now's the time we actually get to prove ourselves to our clients, and now's the time they really need us the most."

Toolmaker Snap-on Inc. expects to spend 35% more on research this year than last year. The company, which has a relationship with Meta Group, has also begun using Forrester Research. The toolmaker has high expectations of its IT research providers, says Al Biland, CIO for the Kenosha, Wis., company. "We don't like to give multiyear, multimillion-dollar contracts to just dabble," he says.

Snap-on added Forrester to help it decide what demographic groups it should use the Internet to reach and where it should spend its advertising dollars. Forrester's initial demographics research for the project was too general, and Snap-on wound up spending extra money for a more complete study focusing on repair technicians and automotive repair shops, Biland says. But he's not complaining--the deeper study proved very valuable. "It was worth every dime we spent with them. They made us see that both approaches to going to market--the Web and the traditional dealer channel--have a lot of relevance," he says.

Meta Group provides Snap-on with advice on technology vendors, Biland says. The firm recommended technology from Onlink, an E-commerce vendor now owned by Siebel Systems Inc., and portfolio-management software from ProSight Inc.--applications that have been important for Snap-on. "You can be like everyone else and install SAP, or you can look for small vendors, and that can be a source of competitive advantage," Biland says. He adds that Onlink's and ProSight's products are designed to meet the particular needs of a business like Snap-on's, not generic applications designed for a wide range of business types. "To get real innovation, you have to look at the fringes," he says.

Most survey respondents say the analyst firms they use provide unbiased advice about products and technology, even though many research firms receive a significant portion of their revenue from IT vendors. Gartner, for instance, gets about 25% of its revenue from vendors, including Oracle, yet last month Gartner issued a report on Oracle that concluded in part that the vendor had lost momentum in the applications market because of buggy software. Oracle posted a rebuttal on its Web site charging Gartner with a "documented bias" against the company in its coverage. Gartner responds that its reports are based on unbiased research of what is happening in the marketplace. "We stand by every piece of research we write," Fleisher says.

Suspicions of favoritism linger. Siri Singh Khalsa, chief technology officer for Healthscribe India Ltd., a medical data-management provider with U.S. headquarters in Sterling, Va., says some research he sees reminds him more of advertising than independent analysis, with a few product weaknesses or problems thrown in "to balance things out a little so you believe the rest of it."

Bias isn't the problem for Ronda Henning, a systems engineer for Harris Corp., a $1.81 billion communications equipment company in Melbourne, Fla. She takes the prejudices she thinks an IT research firm might have in favor of a certain vendor into account when deciding which technology firms to buy from. What's worse, she says, is that some analysts try to gauge the thinking of her company's IT experts and simply affirm that as the company's best strategy. This is a problem, considering that 81% of respondents to our survey look to IT research re-ports not just to validate their IT strategies, but also to set them straight when they're heading in the wrong direction. "They'll play the yes men, and then you don't get what you pay for--independent thinking," Henning says. "That's much more dangerous than having to deal with a bias an analyst has or the firm has." Henning says she guards against that by getting advice from several different firms and by trying to make sure she doesn't give the appearance of leaning toward a particular vendor or technology.

Siemens' Edney doesn't lay the fault for most problems at the door of analyst firms, but at the feet of specific analysts. Some of them just don't do their homework, which is why Edney makes sure to do his own research on a particular analyst before he's ready to work with that person. Siemens looks for analysts with a good two-way flow of information--"someone who's talking to our potential customers to determine what they want and to people like ourselves who are using technology to solve problems," he says. "With analysts, you get a spectrum--you get those running ahead of the game and thinking everything is going to change, and those who have a better feel for the times."

For 11% of survey respondents, the problem isn't that analyst firms are too far ahead, but that they're behind the times. To guard against that, Edney says, it pays for companies to do their own research with customers when they're looking for information on how quickly they'll adopt new technologies. Companies can balance that against research from analyst firms to gain insight into where the technology is headed and how quickly its use will become widespread. Edney says he's not looking for someone to simply validate his IT strategy, and by doing his own research he has a better handle on whether the analysts who disagree with that strategy have valid objections or should be disregarded.

Developing closer relationships with customers is a priority at most businesses, especially in these tight times. And gaining insight into their customers is a key reason more than 60% of respondents to our survey purchase IT research reports. But here's where those ROI concerns kick in: More than a quarter of respondents say IT analysts do a poor or unsatisfactory job helping them understand their customers.

"Money you might spend on analysts to try to give you information about your customers, you could use to take trips to a customer to learn for yourself how they need you," says Mark Ragusa, principal planner for Fujitsu Network Communications in Richardson, Texas. A subsidiary of $45 billion Fujitsu Ltd., Fujitsu Network Communications manufactures fiber-optics and broadband products, and uses analysts mostly for market-trend information. The company primarily relies on Gartner, International Data Corp., and the Yankee Group for research, and generally considers them reliable. But specific results are mixed, Ragusa says, because some analysts aren't on top of issues. "Research is nowhere near as consistently accurate as I would like for it to be," he says. "With analysts, sometimes they hit the nail on the head, and sometimes where they hit is not even the same two-by-four." Rather than cutting back on research spending, the company is focusing more on analysts whose work they have come to respect, Ragusa says.

Analyst firms say they're staying relevant in several ways, from ramping up the number of clients they speak to daily, to sharpening their focus on the projects most in demand, such as deployments with short rollouts and fast return on investment. In short, they're working harder to find ways to help their customers focus on what's right for them, not simply on the next big thing. "People have spent based on analyst predictions and there's no question that's left a bad taste in their mouths," says Meta Group chairman and CEO Dale Kutnick. "Analysts will have to earn the respect and confidence of customers again."

--with Alorie Gilbert

Photo by Anna Curtis