Meet The Elusive Data Scientist

Big Data Talent War: 7 Ways To Win

Nearly three dozen data scientists from academia and business met in Chicago this week to share ideas, discuss challenges, vent frustrations, and generally compare notes about their suddenly high-profile profession.

The meeting, organized by U.K.-based IE, revealed a surprising amount of commonality among the assembled experts, many of whom said they were eager to help their organizations but often run into organizational or infrastructural roadblocks.

"70% of my value is an ability to pull the data, 20% of my value is using data-science methods and asking the right questions, and 10% of my value is knowing the tools," said Catalin Ciobanu, a physicist who spent ten years at Fermi National Accelerator Laboratory (Fermilab) and is now senior manager-BI at Carlson Wagonlit Travel.

[ Data scientists are much in demand, and those who also have communications skills are golden. See Wanted: Qualified Data Scientists, People Skills A Plus. ]

Like a number of the Ph.D.'s at the meeting, Ciobanu suggested a data scientist's chief contribution was an ability to think through complex problems before using high-powered computational systems. "The thought process is the most important ingredient in data science," he said.

This idea was echoed by Scott Nicholson, a Ph.D. in econometrics and the chief data scientist at Accretive Health, which works with hospitals to improve their performance and patient outcomes.

"My definition of a data scientist is someone who uses data to solve problems, end to end, from asking the right questions to making insights actionable," Nicholson told the group.

He said to be successful, a data scientist must be engaged with the business owners and their problems. Otherwise, the data insights won't be iteratively tested and, worse, may be misapplied. Before joining Accretive six months ago, Nicholson ran a group of data scientists at professional social networking site LinkedIn.

Nicholson also spoke for many in the room when he talked about the obstacles, technical and organizational, for conducting data science properly. Obstacles include data sources that are siloed, either because there are fiefdoms inside the organization or because, as is the case in healthcare, there are legal, compliance and privacy rules that prevent data-sharing.

And much of a data scientist's work is what he called the "messy but necessary" business of extracting and cleaning data, getting it ready for use in a model. "It's 70% of the job," he said.

Like their business-side counterparts, the data scientists at the meeting also confirmed the shortage of trained workers.

"For every hundred job openings, there may just be a couple of applicants," said Kirk Borne, professor of astrophysics and computational science at George Mason University, which has graduated 200 Ph.D.'s in the past 20 years. Borne and others also emphasized the need for curiosity coupled with communication skills in the profession.

"Engineering, I think you can pick up," said Accretive's Nicholson. "Curiosity is built in."

One of the most detailed presentations of the day came from Aron Clymer, who manages a team of data scientists for product intelligence at salesforce.com.

The cloud-based back-office giant leverages Clymer's group across its 150 product teams to understand customer behavior and customer sentiment.

The group -- composed of six data scientists and three business analysts -- pulls data from transactions and other feeds found throughout Salesforce.com. Among other data inputs are website interactions, product research, customer support calls, sales visits, customer billing histories and the chatter on social media.

The behavioral data alone amounts to some 1 billion transactions per day, according to Clymer. This data is loaded into a Hadoop database that looks at 500 features.

One of the more interesting ongoing analysis projects at salesforce.com involves the company's Idea Exchange, a customer-facing site that lets the community post feature requests. "We have 26,000 active ideas there," Clymer said. His group mines this resource, conducts sentiment analysis and ships its conclusions to the various product teams. To date, the product teams have been delivering on about four ideas per week based on the analysis, Clymer said.

Salesforce also checks its customers' sentiment on social media, using Radian6, the social media monitoring company it acquired last year. Clymer noted that his strategy is to increasingly use Salesforce.com's own platform for big data and analytic projects as it ramps up its capabilities in these areas.

For instance, one text analytics study using Radian6 revealed that customers were having trouble with their logins. "So in about two weeks we delivered a video on how to reset passwords and issued a patch," Clymer said, adding that this single fix will probably save the company close to $1 million in the course of a year.

In-memory analytics offers subsecond response times and hundreds of thousands of transactions per second. Now falling costs put it in reach of more enterprises. Also in the Analytics Speed Demon special issue of InformationWeek: Louisiana State University hopes to align business and IT more closely through a master's program focused on analytics. (Free registration required.)