Data Scientists: Meet Big Data's Top Guns

Big Data Talent War: 10 Analytics Job Trends

The profession of data scientist is one that requires expertise in multiple technical disciplines, including computer science, analytics, math, modeling, and statistics. It involves analyzing massive data sets for meaningful insights, and then communicating these insights to an organization's management team.

Here's how Anjul Bhambhri, VP of big data products at IBM, describes the profession: "A data scientist is somebody who is inquisitive, who can stare at data and spot trends. It's almost like a Renaissance individual who really wants to learn and bring change to an organization."

Data scientists are often paid a comfortable six-figure salary for their insights, according to Glassdoor.com, and yet employers are struggling to find applicants qualified to analyze their big data streams. According to a 2011 study by the McKinsey Global Institute, the U.S. could face a shortage of up to 190,000 data scientists by 2018.

To gain insight into the world of the data scientist, InformationWeek spoke with a real-life practitioner of the art: Jacob Spoelstra, global head of research and development for Opera Solutions, a predictive analytics firm that employs more than 200 data scientists. A native of South Africa, Spoelstra is based in San Diego. He holds BS and MS degrees in electrical engineering from the University of Pretoria, and a PhD in computer science from the University of Southern California.

What training does a data scientist need? "The skill set we specifically look for is a background in machine learning and modeling, which you get from the computer science, electrical engineering, and statistics fields," said Spoelstra.

[ Government IT faces its own challenges with big data. Read State Agencies Need A Big Data Strategy. ]

Programming ability is essential too. "We often talk about this combination of art and science," said Spoelstra. A talented data scientist must be adept at finding hidden meaning in gigabytes, and often terabytes, of data. It's essential that he or she understand the problem the data may solve.

"What are the behaviors, for instance, that we need to look out for--that we're hoping to find evidence of in the data? For instance, someone losing interest [in a service] might manifest itself in shorter times spent on the Web, and less frequent logins," Spoelstra said.

Data scientists must write programs that capture the concepts they want. "For instance, you may be doing some mathematical transformation where you're looking at a trend over time--the number of logins, or the amount of time (a person) spent on the website. You capture those behaviors using formulas," he added.

Say, for instance, a financial company wants to know which of its customers are most likely to default on a loan in the next 6 to 12 months.

"If you look at somebody's credit bureau file, there could be thousands of raw elements in there," said Spoelstra. "What the data scientist has to do is capture these concepts, which often are not explicitly coded in the data. But the information is there."

And that's where artistic insights can help. The data scientist "must understand the problem from the business' perspective and ask, 'How am I going to find that in the data?'" Spoelstra said.

If the profession pays well, why is there a shortage of data scientists?

"One of the reasons is that there's extremely high demand in the field of big data and predictive analytics," said Spoelstra.

Another problem is that universities don't necessarily provide the skills needed for the job, although Spoelstra believes that may be changing.

Data science is a collaboration between "man and machine," he said.

The human knows the business problem, but the machine can do the grunt work of generating hundreds of thousands of potential useful signals from the data. The machine then searches those signals, and surfaces the ones that may be useful.

Said Spoelstra, "The human is in the loop, and the machine makes the human work better and faster."

See the future of business technology at Interop New York, Oct. 1-5. It's the best place to learn about next-generation technologies including cloud computing, BYOD, big data, and virtualization. Register by Sept. 29 with priority code YLBQNY02 and save up to $150 on passes.