These big data gurus remain in high demand. A hiring side expert talks about what data scientists bring to the table.
Big Data Talent War: 10 Analytics Job Trends
(click image for larger view and for slideshow)
The profession of data scientist is one that requires expertise in multiple technical disciplines, including computer science, analytics, math, modeling, and statistics. It involves analyzing massive data sets for meaningful insights, and then communicating these insights to an organization's management team.
Here's how Anjul Bhambhri, VP of big data products at IBM, describes the profession: "A data scientist is somebody who is inquisitive, who can stare at data and spot trends. It's almost like a Renaissance individual who really wants to learn and bring change to an organization."
Data scientists are often paid a comfortable six-figure salary for their insights, according to Glassdoor.com, and yet employers are struggling to find applicants qualified to analyze their big data streams. According to a 2011 study by the McKinsey Global Institute, the U.S. could face a shortage of up to 190,000 data scientists by 2018.
To gain insight into the world of the data scientist, InformationWeek spoke with a real-life practitioner of the art: Jacob Spoelstra, global head of research and development for Opera Solutions, a predictive analytics firm that employs more than 200 data scientists. A native of South Africa, Spoelstra is based in San Diego. He holds BS and MS degrees in electrical engineering from the University of Pretoria, and a PhD in computer science from the University of Southern California.
What training does a data scientist need? "The skill set we specifically look for is a background in machine learning and modeling, which you get from the computer science, electrical engineering, and statistics fields," said Spoelstra.
Programming ability is essential too. "We often talk about this combination of art and science," said Spoelstra. A talented data scientist must be adept at finding hidden meaning in gigabytes, and often terabytes, of data. It's essential that he or she understand the problem the data may solve.
"What are the behaviors, for instance, that we need to look out for--that we're hoping to find evidence of in the data? For instance, someone losing interest [in a service] might manifest itself in shorter times spent on the Web, and less frequent logins," Spoelstra said.
Data scientists must write programs that capture the concepts they want. "For instance, you may be doing some mathematical transformation where you're looking at a trend over time--the number of logins, or the amount of time (a person) spent on the website. You capture those behaviors using formulas," he added.
Say, for instance, a financial company wants to know which of its customers are most likely to default on a loan in the next 6 to 12 months.
"If you look at somebody's credit bureau file, there could be thousands of raw elements in there," said Spoelstra. "What the data scientist has to do is capture these concepts, which often are not explicitly coded in the data. But the information is there."
And that's where artistic insights can help. The data scientist "must understand the problem from the business' perspective and ask, 'How am I going to find that in the data?'" Spoelstra said.
If the profession pays well, why is there a shortage of data scientists?
"One of the reasons is that there's extremely high demand in the field of big data and predictive analytics," said Spoelstra.
Another problem is that universities don't necessarily provide the skills needed for the job, although Spoelstra believes that may be changing.
Data science is a collaboration between "man and machine," he said.
The human knows the business problem, but the machine can do the grunt work of generating hundreds of thousands of potential useful signals from the data. The machine then searches those signals, and surfaces the ones that may be useful.
Said Spoelstra, "The human is in the loop, and the machine makes the human work better and faster."
See the future of business technology at Interop New York, Oct. 1-5. It's the best place to learn about next-generation technologies including cloud computing, BYOD, big data, and virtualization. Register by Sept. 29 with priority code YLBQNY02 and save up to $150 on passes.
6 Tools to Protect Big DataMost IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Big Data Brings Big Security ProblemsWhy should big data be more difficult to secure? In a word, variety. But the business won’t wait to use it to predict customer behavior, find correlations across disparate data sources, predict fraud or financial risk, and more.
InformationWeek Tech Digest, Nov. 10, 2014Just 30% of respondents to our new survey say their companies are very or extremely effective at identifying critical data and analyzing it to make decisions, down from 42% in 2013. What gives?