Big Data Grows Up: APM Tools Emerging

Big Data: 6 Real-Life Business Cases

Big data infrastructure's creation came out of a need for high performance distributed systems, so it may not be surprising that application performance monitoring tools for Hadoop apps haven't been plentiful -- yet.

As big data platforms start to mature, tools will begin to emerge, and that's just what is expected to happen with Hadoop in the next year or so. As they emerge, more enterprises will take a more serious look at implementing Hadoop.

[Looking for more about tools for big data? Read IBM Wants To Bring More IT To Big Data.]

Concurrent is among the few first movers in this space. Last week the company introduced Driven 2.0, an update to its monitoring and management tool for Hadoop that extends the tool to support Spark, too.

"We are focused on enterprises that are moving their processing to Hadoop or Spark and getting serious about monitoring and management," Concurrent CEO Gary Nakamura told InformationWeek in an interview. Nakamura said that this is still an early market, but maturity is happening rapidly with individual customers evolving from a small need to a large need in a matter of months.

"The systems become resource intensive and chaotic," he said. "We provide visibility."

The Current Market

Management tools for big data are following a similar trajectory as they did for previous infrastructures such as client-server, according to Mike Matchett, senior analyst and consultant for the Taneja Group, a market analyst firm.

"Once I put this into production, you've got multiple jobs, multiple workloads, multiple tenants," Matchett told InformationWeek in an interview. "How do you manage this from an IT perspective? This is IT for Agile DevOps, so the bottom line is someone has got to manage this thing in production."

So far not many companies have offered tools in this space.

"The big guys just haven't been in the game here," Matchett said. "They are just starting to look at it." That makes Concurrent pretty early in the market with a strong tool.

"That's not to say someone else won't come out with another tool in six months. I don't see something here that someone else couldn't develop. But it's just a very complex thing to develop this for a distributed environment" as Concurrent has done with Driven, Matchett said.

APM as Market Driver

He sees tools such as Concurrent's Driven 2.0 as enablers that could spur the big data market.

"I've been in the systems management space long enough to know that people are going to be reluctant to put in systems like Hadoop unless they have tools like Driven to manage it," he said. "They are definitely leading the market right now, but the market is definitely going there."

What are some of the use-cases for this product at the leading edge of the big data infrastructure management market? Nakamura told InformationWeek that one of the most common use-cases involves migrating applications to Hadoop from other platforms.

It's complicated: New frameworks, tools, technologies, and compute engines have created new requirements for DevOps, data operations, and teams. Concurrent Driven 2.0 works with Apache Hive, MapReduce, Cascading, Scalding, and now Spark.

The new Spark support is currently in beta. It provides "the comprehensive performance management required to deliver and maintain production Spark data processes," Concurrent said in a prepared statement announcing Driven 2.0.

Matchett told InformationWeek that many of the tools providing support for big data are likely to merge ultimately into a distributed data center management platform. While the market opportunity may not be huge today, "In six months it will be."