Big Data // Big Data Analytics
News
9/4/2012
10:35 AM
Connect Directly
Google+
RSS
E-Mail
50%
50%
Repost This

Big Data's 'Wild West' Period Stars Hadoop

Will a new generation of powerful visual tools soon hide the complexity of predictive analytics? Pervasive Software bets on it.

These are interesting times for data infrastructure, management, and analytics, says Mike Hoskins, chief technical officer for Pervasive Software, an Austin, Texas-based provider of data management and analytics products.

"We're in a Wild West period here," said Hoskins in a phone interview with InformationWeek. Hadoop is emerging as a big data operating system of sorts for data-intensive analytic workloads. A new generation of powerful visual tools will soon hide the complexity of predictive analytics and help "unleash the power of data science," he predicted.

"Hadoop is a management framework. In some ways, it's already a data operating system for launching and managing hundreds and hundreds of data-intensive programs," Hoskins said.

Big data is high up on Gartner's 2012 Hype Cycle for Emerging Technologies, but growing interest in the topic is warranted. "I've been in software for 30 years, and this is hot for a reason," said Hoskins.

[ Where does Hadoop fit in the data management market? Read Hadoop Won't Displace Data Management Systems: Cloudera. ]

The volumes of big data are exploding--a development that will change aspects of the software industry forever.

"Walmart took 40 years to get their data warehouse at 400 terabytes. Facebook probably generates that every 4 days," said Hoskins. "And Facebook is still only clicks and human interaction. But when you get to machine-generated data, the volume is going to explode wildly again."

The dramatic rise in the use of wireless sensors is another factor.

"We're instrumenting the universe. Everything's a machine, everything is recording at tremendous granular levels," Hoskins said. "There's an opportunity to mine the mountains of machine-generated data with advanced and machine-learning analytics."

Pervasive Software's big data software offerings include DataRush, a platform designed to boost parallel performance of data preparation and analytics tasks, and RushAnalyzer, a visual workflow tool for data access, preparation, analysis, and reporting.

Given Pervasive's big data focus, it's not surprising that Hoskins is bullish on Hadoop, which breaks tradition with the past by not requiring expensive and exotic computer equipment. "Hadoop is a beautiful thing. Its DNA and skeleton are essentially the right architecture and technology for the adoption of commodity hardware," Hoskins said.

Hadoop's distributed architecture offers a distinct advantage over conventional data-management architectures. Hadoop applications, for instance, can run tasks on the node where the data is located. "The big idea of Hadoop is to take the compute to the data," said Hoskins.

The platform, however, has its limitations. Hadoop has security risks, which Hoskins says are being addressed by major proponents of the platform, including Apache, Cloudera, and Hortonworks.

Hadoop can also be tedious and difficult to use. "You're writing low-level Java code, which isn't where people want to be," said Hoskins.

The platform's main weakness is its performance, Hoskins believes, a shortcoming that high-speed dataflow engines like Pervasive's DataRush are designed to address. The next generation of MapReduce--version 2.0 or YARN--will address the framework's performance issues as well.

Hoskins predicts the next 12 to 24 months will bring a wave of powerful visual tools that mask the complexity of parallel programming. "For us to really unleash the power of data science, we must have tooling that lives higher in the stack," said Hoskins. "With tooling, obviously, it becomes easier to find data scientists, because they're operating at a higher level of visual programming, as opposed to low-level Java coding."

The industry-wide shortage of data scientists, however, is a problem that may take longer to resolve. "There's some advanced science here that's not in the province of the traditional business analyst," said Hoskins.

InformationWeek is conducting a survey on big data. Take our InformationWeek 2013 Big Data Survey now. Survey ends Sept. 7.

Comment  | 
Print  | 
More Insights
InformationWeek Elite 100
InformationWeek Elite 100
Our data shows these innovators using digital technology in two key areas: providing better products and cutting costs. Almost half of them expect to introduce a new IT-led product this year, and 46% are using technology to make business processes more efficient.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Elite 100 - 2014
Our InformationWeek Elite 100 issue -- our 26th ranking of technology innovators -- shines a spotlight on businesses that are succeeding because of their digital strategies. We take a close at look at the top five companies in this year's ranking and the eight winners of our Business Innovation awards, and offer 20 great ideas that you can use in your company. We also provide a ranked list of our Elite 100 innovators.
Video
Slideshows
Twitter Feed
Audio Interviews
Archived Audio Interviews
GE is a leader in combining connected devices and advanced analytics in pursuit of practical goals like less downtime, lower operating costs, and higher throughput. At GIO Power & Water, CIO Jim Fowler is part of the team exploring how to apply these techniques to some of the world's essential infrastructure, from power plants to water treatment systems. Join us, and bring your questions, as we talk about what's ahead.