Big Data's 'Wild West' Period Stars Hadoop - InformationWeek
Data Management // Big Data Analytics
10:35 AM
Connect Directly
[Dark Reading Crash Course] Finding & Fixing Application Security Vulnerabilitie
Sep 14, 2017
Hear from a top applications security expert as he discusses key practices for scanning and securi ...Read More>>

Big Data's 'Wild West' Period Stars Hadoop

Will a new generation of powerful visual tools soon hide the complexity of predictive analytics? Pervasive Software bets on it.

These are interesting times for data infrastructure, management, and analytics, says Mike Hoskins, chief technical officer for Pervasive Software, an Austin, Texas-based provider of data management and analytics products.

"We're in a Wild West period here," said Hoskins in a phone interview with InformationWeek. Hadoop is emerging as a big data operating system of sorts for data-intensive analytic workloads. A new generation of powerful visual tools will soon hide the complexity of predictive analytics and help "unleash the power of data science," he predicted.

"Hadoop is a management framework. In some ways, it's already a data operating system for launching and managing hundreds and hundreds of data-intensive programs," Hoskins said.

Big data is high up on Gartner's 2012 Hype Cycle for Emerging Technologies, but growing interest in the topic is warranted. "I've been in software for 30 years, and this is hot for a reason," said Hoskins.

[ Where does Hadoop fit in the data management market? Read Hadoop Won't Displace Data Management Systems: Cloudera. ]

The volumes of big data are exploding--a development that will change aspects of the software industry forever.

"Walmart took 40 years to get their data warehouse at 400 terabytes. Facebook probably generates that every 4 days," said Hoskins. "And Facebook is still only clicks and human interaction. But when you get to machine-generated data, the volume is going to explode wildly again."

The dramatic rise in the use of wireless sensors is another factor.

"We're instrumenting the universe. Everything's a machine, everything is recording at tremendous granular levels," Hoskins said. "There's an opportunity to mine the mountains of machine-generated data with advanced and machine-learning analytics."

Pervasive Software's big data software offerings include DataRush, a platform designed to boost parallel performance of data preparation and analytics tasks, and RushAnalyzer, a visual workflow tool for data access, preparation, analysis, and reporting.

Given Pervasive's big data focus, it's not surprising that Hoskins is bullish on Hadoop, which breaks tradition with the past by not requiring expensive and exotic computer equipment. "Hadoop is a beautiful thing. Its DNA and skeleton are essentially the right architecture and technology for the adoption of commodity hardware," Hoskins said.

Hadoop's distributed architecture offers a distinct advantage over conventional data-management architectures. Hadoop applications, for instance, can run tasks on the node where the data is located. "The big idea of Hadoop is to take the compute to the data," said Hoskins.

The platform, however, has its limitations. Hadoop has security risks, which Hoskins says are being addressed by major proponents of the platform, including Apache, Cloudera, and Hortonworks.

Hadoop can also be tedious and difficult to use. "You're writing low-level Java code, which isn't where people want to be," said Hoskins.

The platform's main weakness is its performance, Hoskins believes, a shortcoming that high-speed dataflow engines like Pervasive's DataRush are designed to address. The next generation of MapReduce--version 2.0 or YARN--will address the framework's performance issues as well.

Hoskins predicts the next 12 to 24 months will bring a wave of powerful visual tools that mask the complexity of parallel programming. "For us to really unleash the power of data science, we must have tooling that lives higher in the stack," said Hoskins. "With tooling, obviously, it becomes easier to find data scientists, because they're operating at a higher level of visual programming, as opposed to low-level Java coding."

The industry-wide shortage of data scientists, however, is a problem that may take longer to resolve. "There's some advanced science here that's not in the province of the traditional business analyst," said Hoskins.

InformationWeek is conducting a survey on big data. Take our InformationWeek 2013 Big Data Survey now. Survey ends Sept. 7.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of IT Report
In today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll