Hadoop Won't Displace Data Management Systems: Cloudera
Big data platform Hadoop will complement, not replace, legacy systems, says Cloudera CEO Mike Olson.
Apache Hadoop is good at many things, but one thing it won't do is conquer the entire data management market. Rather, the big data platform will complement existing enterprise infrastructures and eventually integrate with legacy systems, said Cloudera CEO and founder Mike Olson.
"With almost all emerging technologies, my experience has been that the market tries to pigeonhole the new thing into categories that existing products have defined over the course of some years," said Olson in a phone interview with InformationWeek.
The four-year-old Cloudera is one of the leading proponents of Hadoop, the foundation of the company's Cloudera Enterprise platform of data management products and services. Olson, a 25-year veteran of the relational database industry, was formerly CEO of Sleepycat Software, which Oracle acquired in 2006.
Some Hadoop backers believe the platform's ability to process massive data sets faster than conventional ETL (extract, transform, and load) systems makes it the superior technology. But Olson sees this viewpoint as a misconception of Hadoop's role in the industry.
Hadoop was designed to process new analytic workloads, or capture and retain much larger data volumes than previously possible. As a result, it plays a significant new role in the enterprise's data center.
"Almost without exception, when we see Hadoop in real customer deployments, it's stood up next to existing infrastructure that's aimed at existing business problems," said Olson.
He added, "In my view, it's unlikely that a brand-new entrant to the market is going to displace (legacy) tools for established workloads."
Data that lands in Hadoop needs to move into--and sometimes out of—existing data management systems over the course of its lifetime. And Olson believes this provides an opportunity for Hadoop providers, who must enhance the platform's security and tools.
Hadoop "was built by Internet companies to begin with--companies like Google, Facebook, and Yahoo—and landed entirely inside the firewall on custom infrastructure," he said. "It didn't need to think about security in quite the way that banks and insurance companies do."
Addressing Hadoop's security shortcomings is a major area of effort for Cloudera--as is developing tools that make the platform easier for IT departments to manage.
"You may be a great Oracle [database administrator], or outstanding at keeping SAP running, but you haven't met Hadoop and you're not sure what to do with it," said Olson. "In order for Cloudera Enterprise to take off, it's absolutely critical that we make life easy for data center staff. In addition, analysts and business users should be able to take advantage of the platform. They need great tooling and innovative applications."
Olson points to Cloudera's November 2010 partnership with data integration software provider Informatica as an example of companies working together to merge data from Hadoop into enterprise data management systems.
The sudden emergence of big data has made the database industry a lot more interesting, he said. "I came up in the relational database industry," Olson said. I began working for relational database companies in the middle 1980s as a developer. And I know and love those products. I made a great career in that space."
But "it was pretty boring," he added.
"You made the systems 20% more reliable, and 5% faster every single year, and we charged 5% more every single year. But nothing fundamentally exciting happened," said Olson. "And then all of sudden, in the early-to-middle 2000s, burst these new ideas--NoSQL stores and this hairball Hadoop platform that stored more data and processed it more ways."
Over time, Olson believes, data management platforms will "dwindle in number and increase in capability," as the various systems standardize and unify.
Until then, however, he'll enjoy the big data thrill ride.
"I haven't had this much fun in my career so far," Olson said. "We made a great living, but it wasn't nearly as exciting as it is today."
See the future of business technology at Interop New York Oct. 1-5. It's the best place to learn about next-generation technologies including cloud computing, BYOD, big data, and virtualization. End of Summer Discounts end Sept. 5. Save up to $800 on Interop New York Conference Passes with code WEYLBQNY07.
Google in the Enterprise SurveyThere's no doubt Google has made headway into businesses: Just 28 percent discourage or ban use of its productivity products, and 69 percent cite Google Apps' good or excellent mobility. But progress could still stall: 59 percent of nonusers distrust the security of Google's cloud. Its data privacy is an open question, and 37 percent worry about integration.
CIOs Get Smart About BIIT’s tried for years to simplify business intelligence efforts. Have visual analysis tools and Hadoop and NoSQL databases helped? Respondents to our 2014 InformationWeek Analytics, Business Intelligence, and Information Management Survey have a mixed outlook.