Big Data's 'Wild West' Period Stars Hadoop

These are interesting times for data infrastructure, management, and analytics, says Mike Hoskins, chief technical officer for Pervasive Software, an Austin, Texas-based provider of data management and analytics products.

"We're in a Wild West period here," said Hoskins in a phone interview with InformationWeek. Hadoop is emerging as a big data operating system of sorts for data-intensive analytic workloads. A new generation of powerful visual tools will soon hide the complexity of predictive analytics and help "unleash the power of data science," he predicted.

"Hadoop is a management framework. In some ways, it's already a data operating system for launching and managing hundreds and hundreds of data-intensive programs," Hoskins said.

Big data is high up on Gartner's 2012 Hype Cycle for Emerging Technologies, but growing interest in the topic is warranted. "I've been in software for 30 years, and this is hot for a reason," said Hoskins.

[ Where does Hadoop fit in the data management market? Read Hadoop Won't Displace Data Management Systems: Cloudera. ]

The volumes of big data are exploding--a development that will change aspects of the software industry forever.

"Walmart took 40 years to get their data warehouse at 400 terabytes. Facebook probably generates that every 4 days," said Hoskins. "And Facebook is still only clicks and human interaction. But when you get to machine-generated data, the volume is going to explode wildly again."

The dramatic rise in the use of wireless sensors is another factor.

"We're instrumenting the universe. Everything's a machine, everything is recording at tremendous granular levels," Hoskins said. "There's an opportunity to mine the mountains of machine-generated data with advanced and machine-learning analytics."

Pervasive Software's big data software offerings include DataRush, a platform designed to boost parallel performance of data preparation and analytics tasks, and RushAnalyzer, a visual workflow tool for data access, preparation, analysis, and reporting.

Given Pervasive's big data focus, it's not surprising that Hoskins is bullish on Hadoop, which breaks tradition with the past by not requiring expensive and exotic computer equipment. "Hadoop is a beautiful thing. Its DNA and skeleton are essentially the right architecture and technology for the adoption of commodity hardware," Hoskins said.

Hadoop's distributed architecture offers a distinct advantage over conventional data-management architectures. Hadoop applications, for instance, can run tasks on the node where the data is located. "The big idea of Hadoop is to take the compute to the data," said Hoskins.

The platform, however, has its limitations. Hadoop has security risks, which Hoskins says are being addressed by major proponents of the platform, including Apache, Cloudera, and Hortonworks.

Hadoop can also be tedious and difficult to use. "You're writing low-level Java code, which isn't where people want to be," said Hoskins.

The platform's main weakness is its performance, Hoskins believes, a shortcoming that high-speed dataflow engines like Pervasive's DataRush are designed to address. The next generation of MapReduce--version 2.0 or YARN--will address the framework's performance issues as well.

Hoskins predicts the next 12 to 24 months will bring a wave of powerful visual tools that mask the complexity of parallel programming. "For us to really unleash the power of data science, we must have tooling that lives higher in the stack," said Hoskins. "With tooling, obviously, it becomes easier to find data scientists, because they're operating at a higher level of visual programming, as opposed to low-level Java coding."

The industry-wide shortage of data scientists, however, is a problem that may take longer to resolve. "There's some advanced science here that's not in the province of the traditional business analyst," said Hoskins.

InformationWeek is conducting a survey on big data. Take our InformationWeek 2013 Big Data Survey now. Survey ends Sept. 7.