Data Scientist Shortage: Split Role In Half

"Big Data Creates Big Jobs" trumpeted a recent news headline from research firm Gartner, which predicts that 4.4 million IT jobs will be created globally by 2015 to support big data operations, including 1.9 million IT jobs in the United States.

Sounds great, right? Not necessarily.

"There is not enough talent in the industry. Our public and private education systems are failing us. Therefore, only one-third of the IT jobs will be filled. Data experts will be a scarce, valuable commodity," said Peter Sondergaard, Gartner senior vice president and global head of research, in a statement.

One possible solution to this staffing shortage is to split big data duties in two: data management specialist and data scientist. In a phone interview with InformationWeek, Datalink technology strategist and consulting principal Steve Bulmer said this division of labor would help IT organizations prepare for the coming "tidal wave" of data. "Advanced analytic capabilities are going to be in high demand and hard to find," said Bulmer. "And that's one reason I think that big data -- and the roles that are required to manage and analyze big data -- are going to be split."

[ For another perspective on handling the dearth of big data experts, see Big Data And Analytics Expertise: Beg, Borrow Or Steal? ]

Bulmer is former chief technical officer of Strategic Technologies, or StraTech, a small IT services firm bought in October 2012 by Datalink, which provides data center services to large to medium enterprises. In a recent blog post on big data, Bulmer predicted that the era of the data scientist's "secret domain" is rapidly coming to an end, as organizations try new approaches to manage ever-growing volumes of unstructured information from a variety of sources.

"This is my tsunami warning to IT organizations and providers: IT departments will be called upon more and more to handle the data management phase in order to offload this effort from that scarce data scientist resource; this shift will be the result of concentrating the scientists' focus on the analytics, visualization and actual business absorption of the analytical results," Bulmer wrote.

In fact, the tsunami is just about to hit. In 2016, two-thirds of the mobile workforce will own a smartphone, and global consumers will buy more than 1.6 billion smart mobile devices, Gartner predicts. Those devices, combined with a dramatic increase in machine-to-machine transactions, will mean greater responsibilities for IT staff. "Next year we'll see an awareness from IT professionals that they're going to be called upon to be the information stewards in the big data process," Bulmer told InformationWeek.

The staffing shortages are increasingly easy to spot. When Bulmer attended the Hadoop World trade show in October, for instance, he noticed that the bulletin boards there were packed with postings from startups and other companies seeking data scientists.

Although more business schools are offering analytics courses to provide university graduates with a basic set of data science skills, a looming talent shortage is inevitable.

"The data scientists are going to be lacking for a while, that's for sure," said Bulmer. He cited a recent New York Times article that explains how online retailers are using sophisticated algorithms to change prices hourly to stay ahead of the competition -- projects that demand big-data gurus. "I think more and more data is going to be sought after for competitive advantage," Bulmer said. "And I think it's happening now."

Predictive analysis is getting faster, more accurate and more accessible. Combined with big data, it's driving a new age of experiments. Also in the new, all-digital Advanced Analytics issue of InformationWeek: Are project management offices a waste of money? (Free registration required.)