Will a new generation of powerful visual tools soon hide the complexity of predictive analytics? Pervasive Software bets on it.
These are interesting times for data infrastructure, management, and analytics, says Mike Hoskins, chief technical officer for Pervasive Software, an Austin, Texas-based provider of data management and analytics products.
"We're in a Wild West period here," said Hoskins in a phone interview with InformationWeek. Hadoop is emerging as a big data operating system of sorts for data-intensive analytic workloads. A new generation of powerful visual tools will soon hide the complexity of predictive analytics and help "unleash the power of data science," he predicted.
"Hadoop is a management framework. In some ways, it's already a data operating system for launching and managing hundreds and hundreds of data-intensive programs," Hoskins said.
The volumes of big data are exploding--a development that will change aspects of the software industry forever.
"Walmart took 40 years to get their data warehouse at 400 terabytes. Facebook probably generates that every 4 days," said Hoskins. "And Facebook is still only clicks and human interaction. But when you get to machine-generated data, the volume is going to explode wildly again."
"We're instrumenting the universe. Everything's a machine, everything is recording at tremendous granular levels," Hoskins said. "There's an opportunity to mine the mountains of machine-generated data with advanced and machine-learning analytics."
Pervasive Software's big data software offerings include DataRush, a platform designed to boost parallel performance of data preparation and analytics tasks, and RushAnalyzer, a visual workflow tool for data access, preparation, analysis, and reporting.
Given Pervasive's big data focus, it's not surprising that Hoskins is bullish on Hadoop, which breaks tradition with the past by not requiring expensive and exotic computer equipment. "Hadoop is a beautiful thing. Its DNA and skeleton are essentially the right architecture and technology for the adoption of commodity hardware," Hoskins said.
Hadoop's distributed architecture offers a distinct advantage over conventional data-management architectures. Hadoop applications, for instance, can run tasks on the node where the data is located. "The big idea of Hadoop is to take the compute to the data," said Hoskins.
The platform, however, has its limitations. Hadoop has security risks, which Hoskins says are being addressed by major proponents of the platform, including Apache, Cloudera, and Hortonworks.
Hadoop can also be tedious and difficult to use. "You're writing low-level Java code, which isn't where people want to be," said Hoskins.
The platform's main weakness is its performance, Hoskins believes, a shortcoming that high-speed dataflow engines like Pervasive's DataRush are designed to address. The next generation of MapReduce--version 2.0 or YARN--will address the framework's performance issues as well.
Hoskins predicts the next 12 to 24 months will bring a wave of powerful visual tools that mask the complexity of parallel programming. "For us to really unleash the power of data science, we must have tooling that lives higher in the stack," said Hoskins. "With tooling, obviously, it becomes easier to find data scientists, because they're operating at a higher level of visual programming, as opposed to low-level Java coding."
The industry-wide shortage of data scientists, however, is a problem that may take longer to resolve. "There's some advanced science here that's not in the province of the traditional business analyst," said Hoskins.
6 Tools to Protect Big DataMost IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Big Data Brings Big Security ProblemsWhy should big data be more difficult to secure? In a word, variety. But the business won’t wait to use it to predict customer behavior, find correlations across disparate data sources, predict fraud or financial risk, and more.