Software // Information Management
News
12/11/2013
12:15 PM
Connect Directly
Google+
LinkedIn
Twitter
RSS
E-Mail
50%
50%
Repost This

Splunk Brings Big Data Into Security Monitoring

Telco IDT taps Splunk to optimize IT and isolate security threats. The next step is spotting marketing opportunities hidden in big data.

Machine data tends to come in high volumes, and it's usually a starting point for Splunk, the IT-centric big data analytics platform. Repeating a land-and-expand pattern often experienced by Splunk, customer IDT, a telecommunications and payment services company, has moved from using the platform for datacenter optimization to a high-speed security application. The planned next step is into business and marketing applications.

Splunk monitors server log files and systems data streams and then offers analytics tools to spot and interpret patterns and anomalies in data that are indicative of performance problems or outages. IDT first deployed Splunk in 2008 to address IT systems troubleshooting, and initial success led to wider use across of all of IT. Splunk has effectively replaced a homegrown trouble-analysis application built on a relational database, and IDT reports that the mean time to resolve IT incidents has dropped by more than 20 minutes while network uptime has improved dramatically.

"We had one report that took us 32 hours to run on the old database, and now we're getting it out within two minutes using Splunk," said Golan Ben-Oni, chief security officer and senior VP of network architecture, in a phone interview with InformationWeek. "Once we saw the speed and agility of the platform, it just kind of crept all over the organization."

[Want more on Splunk's Hadoop Integration? Read Splunk Spawns Hunk Hadoop Tool.]

This year, IDT brought Splunk into a security role, replacing a security information and event monitoring system implemented only a year ago. That tool took as long as 15 minutes to issue an alert triggered by correlations of events from security products including Palo Alto Networks firewall software, a FireEye threat-detection platform, and a Fidelis Security Systems network security appliances. Security engineers then took another 15 minutes, at minimum, to isolate infected systems on the network. Between the two delays, IDT was taking too long to respond to security threats.

"It's important to get an infected system off the network as soon as possible, because if a system on the inside of a network has been compromised, it's much easier for an attacker to move laterally within that network because they're behind the firewall," said Ben-Oni.

Taking advantage of Splunk apps, including the Splunk App for Enterprise Security, Splunk App for PCI Compliance, and Splunk App for Palo Alto Networks, IDT was able to speed both alerting and automated response to threats.

"The integration that we did effectively acts on the alerts by triggering Palo Alto to isolate infected systems," Ben-Oni said. "Our target was to get response times down to one minute, but when we implemented it we found the system can react within as little as 18 seconds."

IDT's next planned expansion in the use of Splunk is into detection of business and marketing opportunities, visualizing patterns in data that previously went unobserved.

"We'll be looking at where, geographically, for example, users are coming into the network, so we can identify key emerging market regions that we may need more marketing attention," said Ben-Oni. "A key advantage with Splunk is that we can correlate geographic information from every source, which is something we've exploited on the telecom side of the business to identify carriers that we interconnect with that may be having problems so we could restore services as quickly has possible."

Splunk gathers data both on network usage and on Web site interactions, so another business use case is developing richer behavioral customer profiles, said Ben-Oni.

Now that so much data is being stored in Splunk (which relies on its own, proprietary data store), Ben-Oni says IDT is definitely considering Splunk's "Hunk" integration with Hadoop, which would support high-scale, low-cost storage of data that the company might not otherwise keep.

"I would love to have all my data directly within Splunk's infrastructure, but the beauty of the Hadoop-Splunk integration is that we can, at an extremely low cost, snap up all kinds of data that we wouldn't be able to consider in any other environment," Ben-Oni said.

IT groups need data analytics software that's visual and accessible. Vendors are getting the message. Also in the State Of Analytics issue of InformationWeek: SAP CEO envisions a younger, greener, cloudier company. (Free registration required.)

Doug Henschen is executive editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data, and analytics. He previously served as editor-in-chief of Intelligent Enterprise, editor-in-chief of Transform Magazine, and executive editor at DM News.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Stratustician
50%
50%
Stratustician,
User Rank: Ninja
12/16/2013 | 7:33:26 PM
Big Data for other security tools?
While it's nice to see Big Data being used to process SIEM data as it relates to network activity, I'm curious to see if this model will be applied to more proactive security tools to ideally predict attacks based on unusual behaviour.  For example, while it could be used to isolate a machine that is detected to be compromised, what if it was plugged into IDS/IPS, or even Web Application Firewalls (WAF) to say "Hey, this isn't the normal path that network traffic flows, or these files, or system changes aren't normally used in this way." Theoretically this would help reduce the risk of an infected or compromised machine before it even gets past the firewall.
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
12/11/2013 | 1:28:41 PM
App consolidation play
This is obvoiusly a case of application consolidation within IT. Ben-Oni was cagey about the security monitoring system replaced, other than to say it was a "flagship security information and event monitoring system." The home-grown IT troubleshooting app replaced was built on Oracle database, but I'm guessing the slow search speeds had more to do with limited, home-grown functionality than raw database speeds. 
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
Video
Slideshows
Twitter Feed
Audio Interviews
Archived Audio Interviews
GE is a leader in combining connected devices and advanced analytics in pursuit of practical goals like less downtime, lower operating costs, and higher throughput. At GIO Power & Water, CIO Jim Fowler is part of the team exploring how to apply these techniques to some of the world's essential infrastructure, from power plants to water treatment systems. Join us, and bring your questions, as we talk about what's ahead.