If information is the modern organization's lifeblood (and we happen to think it is), then there's no more critical technology than the software tools that help us consolidate, organize, mine, analyze, and interpret that information--that is, transform raw data into competitive intelligence. No surprise, then, that business intelligence, encompassing everything from data warehousing and mining to performance management, ranks at or near the top of the priority lists of most CIOs.If only we were better at it. A recent survey of 741 IT and business pros by the Data Warehouse Institute found that 81% of respondents still are hamstrung by inaccurate data reporting, 78% by arguments over which data is appropriate for "master data" repositories, and 51% by bad decisions based on incorrect definitions. On the operational side, data warehouses are still too hard to build and manage, especially for companies that can't afford the specialist expertise.

A common measure of a company's BI chops used to be the size of its data warehouse. A decade ago, InformationWeek featured the "1-terabyte club," singling out the likes of AT&T, Bank of America, and Wal-Mart for their plus-sized repositories. Today, the biggest commercial data warehouses massage several hundred terabytes of data, while government outfits like the National Security Agency and NASA's Ames Research Center crunch more than a petabyte (1,000 terabytes) of data.

Still, data warehouse size is no accurate gauge of a company's BI acumen. Even more misleading is when those terabytes of data are found to be scattered across myriad operational and geographic units. In talking about the 500-plus terabytes he inherited in 750 data marts across Hewlett-Packard, CIO Randy Mott speaks for much of the industry when he says: "There was no lack of data. But there was a lack of consistent, timely data spanning different parts of the business."

A survey of large U.S. and U.K. companies by Accenture finds just such a dichotomy: 42% of the 1,009 managers who responded to the survey say they're overloaded with information, yet at the same time they say they can't get the data they need from other departments. What gives?

Silos of structured data are partly to blame, and so it's in many companies' best interests to centralize all of their data--not just blocks for fraud detection, customer acquisition and retention, yield management, and other business priorities--into a true enterprise data warehouse, as Mott espouses in our story this week on HP's internal and commercial plans. But even if all your data is clean, up to date, and easily accessible in a central repository, then there's the matter of all that unstructured content hidden away in Word files, PDFs, spreadsheets, and other electronic documents--the oft-forgotten 90% of a company's information. If you think data warehousing is hard, try getting your arms around enterprise content management.

Meantime, there's plenty of analytical overkill when it comes to the structured stuff. As we reported a few weeks ago, the government's attempt to assemble and mine a massive repository of phone records and other data in order to identify terrorists is being criticized as a monumental waste of money, given the inaccuracy of such macro-level predictive analytics. Wal-Mart, RadioShack, Payless ShoeSource, and other companies are getting some ink for adjusting their workforce schedules on the fly as they crunch data in their new "labor optimization" systems. But can't observant store managers figure out on their own when to staff up and down?

The IT industry and profession are emerging from their adolescent years, but we're still in the infancy of data/content management and business intelligence. As much as we like to hold up the likes of Harrah's and Visa and Wal-Mart as models of BI precision and efficiency, every one of us still has a lot to learn. Our organizations' lives depend on that continued learning.

Rob Preston,
VP/Editor In Chief
[email protected]

To find out more about Rob Preston, please visit his page.