Dashboarding Ourselves 2

The data warehousing/business intelligence (DW/BI) industry is largely self-evaluating — those with the most to gain from favorable reviews are usually also in control of reporting results. Vendors naturally desire to emphasize the most positive and dramatic outcomes, organizations that spend large sums of money are reluctant to publicize failing or underperforming implementations, and analysts walk a fine line between informing their corporate customers and not provoking their vendor patrons. There are some fiercely independent sources of information, such as Nigel Pendse's OLAP Report, but sadly, it only covers a fragment of the market.

In the recent past, some analysts issued reports claiming that extract, transform, and load (ETL) was far superior to alternatives, but ETL vendors sponsored the study. Another study evaluated ROI and TCO for various ETL vendors and, not surprisingly, the ones that demonstrated the best results were also clients of the analyst firm. Many studies employ a shaky methodology, such as the famous Data Warehouse ROI report by IDC (1996 Foundations of Wisdom study), that found an average ROI of 401 percent, but a closer examination revealed that those with ROIs less than zero were excluded from the calculation. A more recent study by TDWI, one of the leading data warehouse education and research organizations, disclosed that almost 70 percent of its survey participants were IT, vendors, or consultants, not stakeholders in organizations who ultimately benefit from these initiatives.

As a DW/BI practitioner, I found these findings very suspicious as they didn't track with my experiences at Hired Brains: They were simply too optimistic and, in some cases, seemed contrived. Despite our best efforts, projects all too often fail to reach their goals for reasons that may not be obvious. It's easy to pinpoint the usual suspects, such as mid-project organization realignment and killer politics, our inability as consultants to convince our clients that certain decisions are suboptimal and a host of others, well documented in the literature (such as TDWI's "Ten Mistakes to Avoid" series). But there are also many cases where everything goes well, yet the initiative never gets traction in the organization, penetration stays at a very low level, and ROI projections aren't met. In many of those cases, this failure to thrive is something of mystery.

To understand how to help our clients achieve success with this technology, we at Hired Brains decided to go to the source and survey the actual BI stakeholders and try to understand the phenomenology of getting BI to work. What we found was very revealing (see Figure 1).

Basically, as an industry, we've mastered the technology, but we've failed at a cognitive level. We haven't been able to change the way people use information, particularly their reliance on standalone spreadsheets and personal databases. (See the sidebar, "In the Shadows.") This single finding not only explains the low penetration rate of BI but also, more ominously, torpedoes any hope of a good ROI, as people only have so much time in a day. If the time spent with desktop tools doesn't decrease, then the benefits from the BI effort won't be realized. We also failed to provide tools that are indispensable or even very useful. The relevance, integration, understanding, and workflow elements of our offerings are all sorely lacking. How can this be?

The Training Mindset

There are a few clues in the survey data, although they're only that; more in-depth research is needed and ongoing. Training clearly has a direct relationship to the perception of success. As the chart shows (see "Training Spend Vs. Outcome" in Figure 1), organizations that devote at least 16 percent of the project budget on training have a greater than 90 percent favorable rating for the data warehouse effort. Organizations that spend 5 percent or less, a pretty typical investment, have a largely unfavorable rating.

FIGURE 1 Data Warehouse dashboard from the user's point of view

Clearly, just spending money on training is insufficient for achieving success, but those organizations that invest in training expose a mindset — cooperation between technology and people and taking an interest in the work that people do — that affects all their decision-making. In those cases where training is given only a few dollars as an afterthought, stakeholders largely find the whole initiative irrelevant. It isn't the training itself that counts, it's what it denotes: An organization that takes an interest in the work that people do and forms a partnership to continuously improve the tools and the output by weaving technology into the work environment, not imposing it.

Why do organizations, then, give training such short shrift? In her landmark book, In the Age of the Smart Machine: The Future of Work and Power, a volume that every practitioner in this business should read, Shoshana Zuboff offers insight into the relationship between learning and control in organizations:

"A commitment to intellective skill development is likely to be hampered when an organization's division of labor continuously replenishes the felt necessity of imperative control. Managers who prove and defend their own legitimacy do not easily share knowledge or engage in inquiry. Workers who feel the requirements of subordinates are not enthusiastic learners. New roles cannot emerge without the structures to support them. If managers are to alter their behavior, then the methods of evaluation and reward that encourage them to do so must be put in place. If employees are to learn to operate in new ways and to broaden their contribution to the life of the business, then career ladders and reward systems reflecting that change must be designed. In this context, access to information is critically important; the structure of access to information expresses the organization's underlying conception of authority."

The correlation between the money spent on training and satisfaction with the system isn't a direct one. Rather, the higher spend level corresponds to organizations that are committed to learning, a crucial component in not only data warehouse success, but also the adoption of technology to change the nature of work in general. The implications are clear: You can't force success with DW/BI without a desire and commitment from organizations to change and improve the flow of information, the optimization of work processes, and the breakdown of artificial barriers that serve certain participants, but not the organization as a whole. This challenge is much greater than "change management," a nebulous term that is applied without rigor; and is obviously a task that is beyond the reach and skill of ordinary data warehousing practitioners.

So the first step is in our court, as an industry, to learn how to bundle the appropriate organizational transformations into the technology implementation. But there's more. Our offerings must be more aligned with the actual work that people do. They have to be more relevant.

Finding Relevancy

For research purposes, Hired Brains defined relevancy simply: Do the tools and information provided by the data warehouse effort provide you a degree of utility great enough to warrant modifying your work processes to incorporate them? A slight majority, 55 percent, said no. Judging by the responses of their clients, most developers don't know what makes a BI environment relevant.

Interestingly, this negative evaluation about relevancy was evenly distributed across organizations that received industry accolades for their efforts and those that didn't. Within organizations, the low marks for relevancy were uniform, so it doesn't appear to be a matter of perspective (analyst vs. executive, for example). The population surveyed was prescreened to ensure that it comprised only "knowledge workers," people who have historically been shown to benefit from DW/BI.

This of course begs the question, "Why are these efforts irrelevant?" Again, the survey offered some clues, but more research is clearly needed. One element that was surprisingly notable was understandability — not to be confused with ease of use, which received surprisingly high marks (although I wonder how something can be irrelevant but easy to use). Part of the problem is clearly the yin and yang of integration. On the one hand, integration is great because a single name is given to a logical element that has different names in different operational systems. On the other hand, most people are familiar with just one of those meanings, so the new term is often a mystery to them. People don't just float out of their smokestacks; they have to be rescued.

Also, what is understandable to a data modeler isn't necessarily understandable to anyone else. In many cases, users are staring at a schema that's loaded with relational and multidimensional terms, such as keys, joins, dimensions, attributes, and slices. With all the airtime about metadata in this industry, why is this still so needlessly confusing? For the most part, metadata is still a technical issue and very little good has trickled down to users.

Part of the answer is that, despite good intentions, some organizations may find it impossible to make DW/BI relevant. Developing a spreadsheet or personal database is a singular effort and in those organizations described by Zuboff in her book, the collaboration needed to make BI successful is just not possible. A collection of singular efforts, inefficient and potentially inaccurate as it is, simply has a greater chance of being used as it skirts the lines of authority and control.

Another less ominous, but still dysfunctional problem, is that it's simply too difficult to actually build models in most BI tools, which are primarily designed for ad hoc query and analysis using prebuilt relationships. Most BI tools (with some exceptions) don't let users save data. All these are features available in a spreadsheet, overshadowing the benefits of staged, integrated, cleansed data through a DW/BI environment.

Work to Do

Training, and to a larger extent, the mentality of the organization toward learning, is a key indicator of potential DW/BI success. Equally important is the need to deliver tools — in an atmosphere of learning and cooperation — that have a high degree of relevance to the work people do, something that is often trivialized by IT. There's a lot of work to do, but one thing I've learned from this research is that stepping back a little from DW/BI technology reveals a very complicated landscape strewn with hazards. Our industry — encompassing vendors, analysts, journalists, and practitioners — is ill-equipped for dealing with the challenges of making data warehousing and business intelligence successful. We need to strengthen our practice portfolios and partner with our clients in order to implement programs that encompass technology as well as organizational development.

And it wouldn't hurt to beef up the usefulness of metadata, either.

Neil Raden is the founder of Hired Brains, a consulting, systems integration, and implementation services firm that also provides market research and advisory services. He is also a consultant, author, and speaker on BI, data warehousing, and IT strategy.