Big Data // Big Data Analytics
News
8/30/2012
05:01 PM
Connect Directly
Google+
RSS
E-Mail
50%
50%
Repost This

Data Scientists: Meet Big Data's Top Guns

These big data gurus remain in high demand. A hiring side expert talks about what data scientists bring to the table.

Big Data Talent War: 10 Analytics Job Trends
Big Data Talent War: 10 Analytics Job Trends
(click image for larger view and for slideshow)

The profession of data scientist is one that requires expertise in multiple technical disciplines, including computer science, analytics, math, modeling, and statistics. It involves analyzing massive data sets for meaningful insights, and then communicating these insights to an organization's management team.

Here's how Anjul Bhambhri, VP of big data products at IBM, describes the profession: "A data scientist is somebody who is inquisitive, who can stare at data and spot trends. It's almost like a Renaissance individual who really wants to learn and bring change to an organization."

Data scientists are often paid a comfortable six-figure salary for their insights, according to Glassdoor.com, and yet employers are struggling to find applicants qualified to analyze their big data streams. According to a 2011 study by the McKinsey Global Institute, the U.S. could face a shortage of up to 190,000 data scientists by 2018.

To gain insight into the world of the data scientist, InformationWeek spoke with a real-life practitioner of the art: Jacob Spoelstra, global head of research and development for Opera Solutions, a predictive analytics firm that employs more than 200 data scientists. A native of South Africa, Spoelstra is based in San Diego. He holds BS and MS degrees in electrical engineering from the University of Pretoria, and a PhD in computer science from the University of Southern California.

What training does a data scientist need? "The skill set we specifically look for is a background in machine learning and modeling, which you get from the computer science, electrical engineering, and statistics fields," said Spoelstra.

[ Government IT faces its own challenges with big data. Read State Agencies Need A Big Data Strategy. ]

Programming ability is essential too. "We often talk about this combination of art and science," said Spoelstra. A talented data scientist must be adept at finding hidden meaning in gigabytes, and often terabytes, of data. It's essential that he or she understand the problem the data may solve.

"What are the behaviors, for instance, that we need to look out for--that we're hoping to find evidence of in the data? For instance, someone losing interest [in a service] might manifest itself in shorter times spent on the Web, and less frequent logins," Spoelstra said.

Data scientists must write programs that capture the concepts they want. "For instance, you may be doing some mathematical transformation where you're looking at a trend over time--the number of logins, or the amount of time (a person) spent on the website. You capture those behaviors using formulas," he added.

Say, for instance, a financial company wants to know which of its customers are most likely to default on a loan in the next 6 to 12 months.

"If you look at somebody's credit bureau file, there could be thousands of raw elements in there," said Spoelstra. "What the data scientist has to do is capture these concepts, which often are not explicitly coded in the data. But the information is there."

And that's where artistic insights can help. The data scientist "must understand the problem from the business' perspective and ask, 'How am I going to find that in the data?'" Spoelstra said.

If the profession pays well, why is there a shortage of data scientists?

"One of the reasons is that there's extremely high demand in the field of big data and predictive analytics," said Spoelstra.

Another problem is that universities don't necessarily provide the skills needed for the job, although Spoelstra believes that may be changing.

Data science is a collaboration between "man and machine," he said.

The human knows the business problem, but the machine can do the grunt work of generating hundreds of thousands of potential useful signals from the data. The machine then searches those signals, and surfaces the ones that may be useful.

Said Spoelstra, "The human is in the loop, and the machine makes the human work better and faster."

See the future of business technology at Interop New York, Oct. 1-5. It's the best place to learn about next-generation technologies including cloud computing, BYOD, big data, and virtualization. Register by Sept. 29 with priority code YLBQNY02 and save up to $150 on passes.

Comment  | 
Print  | 
More Insights
InformationWeek Elite 100
InformationWeek Elite 100
Our data shows these innovators using digital technology in two key areas: providing better products and cutting costs. Almost half of them expect to introduce a new IT-led product this year, and 46% are using technology to make business processes more efficient.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Elite 100 - 2014
Our InformationWeek Elite 100 issue -- our 26th ranking of technology innovators -- shines a spotlight on businesses that are succeeding because of their digital strategies. We take a close at look at the top five companies in this year's ranking and the eight winners of our Business Innovation awards, and offer 20 great ideas that you can use in your company. We also provide a ranked list of our Elite 100 innovators.
Video
Slideshows
Twitter Feed
Audio Interviews
Archived Audio Interviews
GE is a leader in combining connected devices and advanced analytics in pursuit of practical goals like less downtime, lower operating costs, and higher throughput. At GIO Power & Water, CIO Jim Fowler is part of the team exploring how to apply these techniques to some of the world's essential infrastructure, from power plants to water treatment systems. Join us, and bring your questions, as we talk about what's ahead.