Big Data Opens New Doors For Financial Analysts

Big data is changing how lot of businesses operate, especially in the healthcare, industrial, and retail sectors, to name just a few. Finance is another area where big data is making its mark, as emerging technologies such as Hadoop, NoSQL and Storm allow investment gurus to analyze non-traditional data sets that previously weren't accessible.

That's according to Rod Bodkin, founder and CEO of Think Big Analytics, a 3-year-old big data consultancy that has worked with many large corporations, including Amazon, EMC, Facebook and Intel, to help resolve their data management dilemmas. Before launching Think Big Analytics, Bodkin was VP of engineering at Quantcast, where he managed the data science and engineering staff. And back in the 90's, he was cofounder and CTO of C-Bridge, a B2B applications provider.

In a phone interview with InformationWeek, Bodkin said that big data can help global money managers analyze information in new and innovative ways.

[ How can big data help boost the bottom line? Read Big Data's New Fan: Your CFO. ]

"We think it's already having a big impact," said Bodkin of big data's role in the financial industry. "When you look at capital markets, you see that there's a lot of opportunity to apply new data sets and new algorithms. Fundamentally, we see big data coming in and letting people solve problems that were never before solvable."

Such as?

"We see things like being able to run live simulations of trading strategies and incorporate non-traditional data sets -- like joining together order and trade data with satellite imagery or geolocation data, or sentiment data from social networks," said Bodkin.

Take social media streams, for instance. Stock traders can analyze Twitter or other social posts to gauge public sentiment on a specific company or industry. "We have customers that are incorporating that kind of information," said Bodkin. "Now, a lot of times it's not as simple as a direct correlation. You want to see a combination of sentiment plus some other factors like fundamentals, trading activity, and trending over time."

Sentiment analysis of social media streams is gaining popularity among retailers as well. "We see big data being important in retail operations in terms of building engagement with consumers and getting to know the customer better, building intimacy," said Bodkin.

Not every big data effort pays off, naturally, and there's usually a fair amount of trial and error involved.

"With the big data approach, you're able to test out a lot of new variables and see what's interesting," Bodkin noted. "Lots of the experiments fail, but the ones that succeed can produce tremendous returns. They empower smart analysts who are working on problems … to be a lot more innovative and try out many different things."

During his tenure at Quantcast, Bodkin helped pioneer the use of Hadoop and NoSQL for batch and real-time analysis. He says these big data technologies, working in conjunction with event-stream processing systems like Storm, are already playing a key role in financial market operations.

"Put together, you end up with a very interesting … capability to capture and analyze data, build predictive models, and back test and simulate them," said Bodkin. "We see Hadoop coming in as a real asset to augment that, letting you store more diverse data in a raw form, so you can store the kind of text, social and rich document data that was not easily handled in traditional databases. We see it extending, complimenting and ultimately changing the way companies deal with analytic data sets."

Making decisions based on flashy macro trends while ignoring "little data" fundamentals is a recipe for failure. Also in the new, all-digital Blinded By Big Data issue of InformationWeek: How Coke Bottling's CIO manages mobile strategy. (Free registration required.)