Data Management // Big Data Analytics
10:35 AM
Connect Directly

Big Data's 'Wild West' Period Stars Hadoop

Will a new generation of powerful visual tools soon hide the complexity of predictive analytics? Pervasive Software bets on it.

These are interesting times for data infrastructure, management, and analytics, says Mike Hoskins, chief technical officer for Pervasive Software, an Austin, Texas-based provider of data management and analytics products.

"We're in a Wild West period here," said Hoskins in a phone interview with InformationWeek. Hadoop is emerging as a big data operating system of sorts for data-intensive analytic workloads. A new generation of powerful visual tools will soon hide the complexity of predictive analytics and help "unleash the power of data science," he predicted.

"Hadoop is a management framework. In some ways, it's already a data operating system for launching and managing hundreds and hundreds of data-intensive programs," Hoskins said.

Big data is high up on Gartner's 2012 Hype Cycle for Emerging Technologies, but growing interest in the topic is warranted. "I've been in software for 30 years, and this is hot for a reason," said Hoskins.

[ Where does Hadoop fit in the data management market? Read Hadoop Won't Displace Data Management Systems: Cloudera. ]

The volumes of big data are exploding--a development that will change aspects of the software industry forever.

"Walmart took 40 years to get their data warehouse at 400 terabytes. Facebook probably generates that every 4 days," said Hoskins. "And Facebook is still only clicks and human interaction. But when you get to machine-generated data, the volume is going to explode wildly again."

The dramatic rise in the use of wireless sensors is another factor.

"We're instrumenting the universe. Everything's a machine, everything is recording at tremendous granular levels," Hoskins said. "There's an opportunity to mine the mountains of machine-generated data with advanced and machine-learning analytics."

Pervasive Software's big data software offerings include DataRush, a platform designed to boost parallel performance of data preparation and analytics tasks, and RushAnalyzer, a visual workflow tool for data access, preparation, analysis, and reporting.

Given Pervasive's big data focus, it's not surprising that Hoskins is bullish on Hadoop, which breaks tradition with the past by not requiring expensive and exotic computer equipment. "Hadoop is a beautiful thing. Its DNA and skeleton are essentially the right architecture and technology for the adoption of commodity hardware," Hoskins said.

Hadoop's distributed architecture offers a distinct advantage over conventional data-management architectures. Hadoop applications, for instance, can run tasks on the node where the data is located. "The big idea of Hadoop is to take the compute to the data," said Hoskins.

The platform, however, has its limitations. Hadoop has security risks, which Hoskins says are being addressed by major proponents of the platform, including Apache, Cloudera, and Hortonworks.

Hadoop can also be tedious and difficult to use. "You're writing low-level Java code, which isn't where people want to be," said Hoskins.

The platform's main weakness is its performance, Hoskins believes, a shortcoming that high-speed dataflow engines like Pervasive's DataRush are designed to address. The next generation of MapReduce--version 2.0 or YARN--will address the framework's performance issues as well.

Hoskins predicts the next 12 to 24 months will bring a wave of powerful visual tools that mask the complexity of parallel programming. "For us to really unleash the power of data science, we must have tooling that lives higher in the stack," said Hoskins. "With tooling, obviously, it becomes easier to find data scientists, because they're operating at a higher level of visual programming, as opposed to low-level Java coding."

The industry-wide shortage of data scientists, however, is a problem that may take longer to resolve. "There's some advanced science here that's not in the province of the traditional business analyst," said Hoskins.

InformationWeek is conducting a survey on big data. Take our InformationWeek 2013 Big Data Survey now. Survey ends Sept. 7.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
6 Tools to Protect Big Data
6 Tools to Protect Big Data
Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Register for InformationWeek Newsletters
White Papers
Current Issue
Top IT Trends to Watch in Financial Services
IT pros at banks, investment houses, insurance companies, and other financial services organizations are focused on a range of issues, from peer-to-peer lending to cybersecurity to performance, agility, and compliance. It all matters.
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on for the week of October 9, 2016. We'll be talking with the editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll