Hadoop Won't Displace Data Management Systems: Cloudera

Apache Hadoop is good at many things, but one thing it won't do is conquer the entire data management market. Rather, the big data platform will complement existing enterprise infrastructures and eventually integrate with legacy systems, said Cloudera CEO and founder Mike Olson.

"With almost all emerging technologies, my experience has been that the market tries to pigeonhole the new thing into categories that existing products have defined over the course of some years," said Olson in a phone interview with InformationWeek.

The four-year-old Cloudera is one of the leading proponents of Hadoop, the foundation of the company's Cloudera Enterprise platform of data management products and services. Olson, a 25-year veteran of the relational database industry, was formerly CEO of Sleepycat Software, which Oracle acquired in 2006.

Some Hadoop backers believe the platform's ability to process massive data sets faster than conventional ETL (extract, transform, and load) systems makes it the superior technology. But Olson sees this viewpoint as a misconception of Hadoop's role in the industry.

Hadoop was designed to process new analytic workloads, or capture and retain much larger data volumes than previously possible. As a result, it plays a significant new role in the enterprise's data center.

[ What does it take to be a data scientist? Read Data Scientists: Meet Big Data's Top Guns. ]

"Almost without exception, when we see Hadoop in real customer deployments, it's stood up next to existing infrastructure that's aimed at existing business problems," said Olson.

He added, "In my view, it's unlikely that a brand-new entrant to the market is going to displace (legacy) tools for established workloads."

Data that lands in Hadoop needs to move into--and sometimes out of—existing data management systems over the course of its lifetime. And Olson believes this provides an opportunity for Hadoop providers, who must enhance the platform's security and tools.

Hadoop "was built by Internet companies to begin with--companies like Google, Facebook, and Yahoo—and landed entirely inside the firewall on custom infrastructure," he said. "It didn't need to think about security in quite the way that banks and insurance companies do."

Addressing Hadoop's security shortcomings is a major area of effort for Cloudera--as is developing tools that make the platform easier for IT departments to manage.

"You may be a great Oracle [database administrator], or outstanding at keeping SAP running, but you haven't met Hadoop and you're not sure what to do with it," said Olson. "In order for Cloudera Enterprise to take off, it's absolutely critical that we make life easy for data center staff. In addition, analysts and business users should be able to take advantage of the platform. They need great tooling and innovative applications."

Olson points to Cloudera's November 2010 partnership with data integration software provider Informatica as an example of companies working together to merge data from Hadoop into enterprise data management systems.

The sudden emergence of big data has made the database industry a lot more interesting, he said. "I came up in the relational database industry," Olson said. I began working for relational database companies in the middle 1980s as a developer. And I know and love those products. I made a great career in that space."

But "it was pretty boring," he added.

"You made the systems 20% more reliable, and 5% faster every single year, and we charged 5% more every single year. But nothing fundamentally exciting happened," said Olson. "And then all of sudden, in the early-to-middle 2000s, burst these new ideas--NoSQL stores and this hairball Hadoop platform that stored more data and processed it more ways."

Over time, Olson believes, data management platforms will "dwindle in number and increase in capability," as the various systems standardize and unify.

Until then, however, he'll enjoy the big data thrill ride.

"I haven't had this much fun in my career so far," Olson said. "We made a great living, but it wasn't nearly as exciting as it is today."

See the future of business technology at Interop New York Oct. 1-5. It's the best place to learn about next-generation technologies including cloud computing, BYOD, big data, and virtualization. End of Summer Discounts end Sept. 5. Save up to $800 on Interop New York Conference Passes with code WEYLBQNY07.