Splunk App For Stream Adds Real-Time Wire Data

Location Analytics + Maps: 10 Eureka Moments

Splunk deployments got a boost on Tuesday with the release of the Splunk App for Stream. The new app enables real-time collection of wire data, or machine data sent between systems across a network, which can then be monitored, analyzed, and stored for performance, security, and other big data purposes.

"The Splunk App for Stream is a very important step forward for Splunk," said Jim Frey, VP of network management research at Enterprise Management Associates, in an email interview. "The company has relied on partners to gather and analyze packet data from the network, and with this new [release] it can offer a complete solution entirely on its own."

Splunk App for Stream, which is available as a free download to Splunk Enterprise and Splunk Cloud deployments, is the first new product born out of Splunk's 2013 acquisition of Cloudmeter. Frey described it as a plus for existing Splunk deployments because it will provide additional visibility into IT operational health.

"This is important because packet inspection can reveal a broad range of quality and activity insights that are not available from other monitoring data sources," Frey said. "Splunk App for Stream will not replace the need for higher-end packet-based performance monitoring solutions, but it will help to significantly advance the value and effectiveness of Splunk deployments, particularly for operational and security monitoring purposes."

[What's the key to big data success? Here's a clue: Start small. Read 8 Reasons Big Data Projects Fail.]

A day earlier, ExtraHop, a Splunk partner, announced new iterations of its Open Data Stream platform that support wire data streams to MongoDB and Elasticsearch environments. ExtraHop also offers Open Data Stream versions for Splunk and VMware Log Insight.

Frey called the new additions "extremely positive steps forward," adding that from an anecdotal perspective, enterprise IT demand for ExtraHop's open data approach is in its early stages and poised for significant growth.

"Application and business activity performance and quality measure have always been available by examining the streams of packets that cross the network. Network and application operations teams have known this for a long time, but rarely have taken full advantage outside of the troubleshooting and support context," Frey said. "ExtraHop is helping make the leap for them, by allowing those higher-level indicators to be culled from live monitoring and sent off to open big data stores for analysis."

{image 1}

ExtraHop preaches its open data approach as a means of avoiding vendor lock-in and data silos that can hamstring big data initiatives and introduce new headaches into an already complicated IT environment. The open source MongoDB and Elasticsearch platforms fit the bill both in terms of shared mindset and real-world demand, according to Jesse Rothstein, ExtraHop's CEO and co-founder.

"We chose to launch our Open Data Stream capabilities with MongoDB and Elasticsearch due to a philosophical belief that data should not be locked in restrictive silos, as well as significant market pull from our customer base," Rothstein said in an email. "Our customers are using these non-proprietary datastores for large-scale, multidimensional analysis and have requested native integration."

Expect additional Open Data Stream rollouts down the line. "Our vision for setting IT data free is by no means achieved with just these relationships with MongoDB and Elasticsearch," Rothstein said. "While a great start, we recognize that ExtraHop wire data can be utilized in a broader range of applications moving forward."

Indeed, the IT complexity involved in big data makes it likely there will be more crisscrossing of proprietary and open-source tools for collecting, monitoring, analyzing, storing, and securing information, especially with potentially massive growth of machine data as the Internet of Things takes off.

"There are many use cases to be addressed and most deployments are complementary. However, big data solutions of all stripes are evolving and expanding," Frey of EMA said. "In my view, there will be more and more overlap in the not-so-distant future."

When selecting servers to support analytics, consider data center capacity, storage, and computational intensity. Get the new Hadoop Hardware: One Size Doesn't Fit All issue of InformationWeek Tech Digest today (free registration required).