For more than a decade, big companies and sophisticated data aggregators have adopted data warehouses, yet few have mastered them, and many have outright failed in the effort or have been scared off by the complexity. The goal is to give workers access to real-time data across departments and geographic units, but more often than not, data warehouses end up as costly clunkers with outdated, inconsistent, and missing information.

Hewlett-Packard thinks it can help companies--and not just the biggest and most sophisticated ones--do a lot better. Over the past few months, HP has tiptoed into the data warehousing market with a system, called Neoview, born in its research labs and built first for its internal use.

Until sitting down with InformationWeek recently, the company has been mum on the initiative--not so much as a peep from its normally talkative marketing team. Indeed, it's an unlikely move into a sector where IBM, Oracle, SAS Institute, and Teradata have years of experience, well regarded products, and loyal customers. Those four vendors--along with Microsoft, which has muscled in on the strength of its SQL Server database--hold about 85% of the $5.2 billion-a-year data warehousing software market, a sector IDC projects will grow 9.5% annually through 2010.

CIO Mott is HP's data warehousing guru -- and guinea pig Photo by Sacha Lecca

HP's chance of breaking into this established market would seem slim, but a couple of advantages are working in its favor: namely, two of data warehousing's most knowledgeable figures, CEO Mark Hurd and CIO Randy Mott.

Before joining HP in March 2005, Hurd ran NCR and its Teradata division, giving him an insider's understanding of the arcane world of data extraction, algorithms, and database "table joins." Mott has been the country's highest-profile data warehouse user, having built and managed huge Teradata-based systems as Wal-Mart's CIO in the 1990s, then as CIO at Dell earlier this decade.

Teradata in particular seems vulnerable to HP's push. No outsiders know Teradata's products or strategy better than Hurd and Mott, who talk up Neoview when visiting customers--including Teradata users.

EATING ITS DOG FOOD
Neoview's proving ground will be HP's own data centers. HP's IT organization fired up an enterprise data warehouse last May as part of a three-year overhaul of the company's IT operations, put in motion shortly after Mott joined the company in July 2005. That larger project involves reducing the number of applications used within HP from 5,000 to 1,500 and consolidating 85 data centers to six.

The new data warehouse must evolve in lockstep with that wider initiative. It already has 180 terabytes of raw data and 75 terabytes of "usable" data. By 2008, it will be at least twice that size. Some 50,000 employees, a third of HP's global workforce, will have access to it. Eventually, HP's suppliers, distributors, and business customers will too, Mott says. A stickler for deadlines and accountability, Mott is managing HP's three-year IT overhaul and joined-at-the-hip data warehouse with the hands-on attention Ike gave to D-Day.

But Mott and his staff didn't take the safe route in choosing the underlying technology. Rather than go back to the Teradata platform Hurd and Mott know so well, the company is gambling on technology from its acquisition portfolio: the Tandem NonStop operating system and database.

Tandem Computers built a multibillion-dollar business around NonStop in the 1980s and '90s before being acquired by Compaq Computer in 1997, which was itself acquired by HP in 2002. NonStop has a sterling reputation for transaction processing, but it's unproven as the cerebral cortex of a business intelligence environment, where sorting and joining large tables of data require a different feature set (see story, Nonstop Keeps Moving Along).

HP engineers had been tinkering with the NonStop software with that in mind before the arrival of Hurd and Mott, but it wasn't until Mott's organization gave its stamp of approval that HP decided to forge ahead with a commercial product. "We had a very strong influence on their overall road map," Mott says. (Full disclosure: Mott is a member of InformationWeek's editorial advisory board, a relationship that had no bearing on this story.)

Neoview was conceived as a data warehousing appliance--similar to those sold by Netezza and others--then morphed into a high-end system. A few pages on HP's Web site describe the product line, though there's been no formal unveiling and HP's strategy remains mostly hearsay in the market. The line consists of the NonStop OS microkernel and database, HP Integrity servers and StorageWorks storage system, a management dashboard for monitoring system performance, and the ability to extract data from line-of-business databases and load it into the warehouse.

That's most of what you need to build a large-scale data warehouse, with one important exception: tools for data analysis. For those, HP is partnering with BI specialists, including Business Objects, Cognos, Hyperion, Informatica, MicroStrategy, and SAS. HP custom developed Java-based BI reporting tools for its internal rollout, but it has no plans to commercialize them. That part of the market could prove hard to resist; IDC pegs the data warehousing tools business at $9.6 billion a year, even bigger than the database portion.