Big Data // Big Data Analytics
News
8/30/2012
05:01 PM
Connect Directly
Google+
RSS
E-Mail
50%
50%

Data Scientists: Meet Big Data's Top Guns

These big data gurus remain in high demand. A hiring side expert talks about what data scientists bring to the table.

Big Data Talent War: 10 Analytics Job Trends
Big Data Talent War: 10 Analytics Job Trends
(click image for larger view and for slideshow)

The profession of data scientist is one that requires expertise in multiple technical disciplines, including computer science, analytics, math, modeling, and statistics. It involves analyzing massive data sets for meaningful insights, and then communicating these insights to an organization's management team.

Here's how Anjul Bhambhri, VP of big data products at IBM, describes the profession: "A data scientist is somebody who is inquisitive, who can stare at data and spot trends. It's almost like a Renaissance individual who really wants to learn and bring change to an organization."

Data scientists are often paid a comfortable six-figure salary for their insights, according to Glassdoor.com, and yet employers are struggling to find applicants qualified to analyze their big data streams. According to a 2011 study by the McKinsey Global Institute, the U.S. could face a shortage of up to 190,000 data scientists by 2018.

To gain insight into the world of the data scientist, InformationWeek spoke with a real-life practitioner of the art: Jacob Spoelstra, global head of research and development for Opera Solutions, a predictive analytics firm that employs more than 200 data scientists. A native of South Africa, Spoelstra is based in San Diego. He holds BS and MS degrees in electrical engineering from the University of Pretoria, and a PhD in computer science from the University of Southern California.

What training does a data scientist need? "The skill set we specifically look for is a background in machine learning and modeling, which you get from the computer science, electrical engineering, and statistics fields," said Spoelstra.

[ Government IT faces its own challenges with big data. Read State Agencies Need A Big Data Strategy. ]

Programming ability is essential too. "We often talk about this combination of art and science," said Spoelstra. A talented data scientist must be adept at finding hidden meaning in gigabytes, and often terabytes, of data. It's essential that he or she understand the problem the data may solve.

"What are the behaviors, for instance, that we need to look out for--that we're hoping to find evidence of in the data? For instance, someone losing interest [in a service] might manifest itself in shorter times spent on the Web, and less frequent logins," Spoelstra said.

Data scientists must write programs that capture the concepts they want. "For instance, you may be doing some mathematical transformation where you're looking at a trend over time--the number of logins, or the amount of time (a person) spent on the website. You capture those behaviors using formulas," he added.

Say, for instance, a financial company wants to know which of its customers are most likely to default on a loan in the next 6 to 12 months.

"If you look at somebody's credit bureau file, there could be thousands of raw elements in there," said Spoelstra. "What the data scientist has to do is capture these concepts, which often are not explicitly coded in the data. But the information is there."

And that's where artistic insights can help. The data scientist "must understand the problem from the business' perspective and ask, 'How am I going to find that in the data?'" Spoelstra said.

If the profession pays well, why is there a shortage of data scientists?

"One of the reasons is that there's extremely high demand in the field of big data and predictive analytics," said Spoelstra.

Another problem is that universities don't necessarily provide the skills needed for the job, although Spoelstra believes that may be changing.

Data science is a collaboration between "man and machine," he said.

The human knows the business problem, but the machine can do the grunt work of generating hundreds of thousands of potential useful signals from the data. The machine then searches those signals, and surfaces the ones that may be useful.

Said Spoelstra, "The human is in the loop, and the machine makes the human work better and faster."

See the future of business technology at Interop New York, Oct. 1-5. It's the best place to learn about next-generation technologies including cloud computing, BYOD, big data, and virtualization. Register by Sept. 29 with priority code YLBQNY02 and save up to $150 on passes.

Comment  | 
Print  | 
More Insights
6 Tools to Protect Big Data
6 Tools to Protect Big Data
Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest - July 22, 2014
Sophisticated attacks demand real-time risk management and continuous monitoring. Here's how federal agencies are meeting that challenge.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
A UBM Tech Radio episode on the changing economics of Flash storage used in data tiering -- sponsored by Dell.
Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.