Big Data // Software Platforms
News
8/25/2014
12:05 PM
Connect Directly
LinkedIn
Twitter
RSS
E-Mail
50%
50%

Analytics For All, No Data Scientists Needed

While there's a role for Ph.D.-level experts, the real power is in making advanced analysis work for mainstream -- often Excel-wielding -- business users. Here's how.

Download the entire new issue of InformationWeek Tech Digest, distributed in an all-digital format (free registration required).

Data, data everywhere, and nary a data scientist in sight. Or at least, not one you can afford. It's a classic Catch-22. To thrive, businesses need to pull financial, sales, predictive, social, and other data into a complete view of the customer. But big data practitioners with fancy degrees who can bring sophisticated analytics chops to bear on that effort start in the six figures, if you can even find one.

Academics and consultants pontificate on the crisis. McKinsey & Co. exclaims that advanced big data analytics, driven partly by the Internet of Things, could increase GDP in retailing and manufacturing by up to $325 billion annually and trim nearly as much from the cost of healthcare and government services by 2020. Too bad most organizations will never be able to hire that expertise. Yep, the world's got big data envy bad, and a data scientist is the silver bullet we all need.

Here's an alternative viewpoint: You don't need them. Instead, bring big data analytics down to earth, train some people, and use the tools you have, with a few select additions. Now before you go all Pi and post N∞ comments opposing the concept, hear me out.

Yes there's an explosion of data. Total digital data is doubling in size every two years, according to the annual IDC Digital Universe Survey. IBM says 90% of the data in the world today has been created in the last two years, and we're certainly not slowing down. But guess what? There's also an explosion of tools and tactics to help knowledge workers tap into and make sense of it all. Incumbent enterprise data management vendors such as IBM, Informatica, SAP, and Oracle have beefed up their suites through development and acquisition. The cloud provides ample room to store all the data you can afford to collect, and Amazon and Google have added query tools and engines to use it. Add in the usual startups and specialists, like Domo, Pentaho, Tableau, and Tibco/Jaspersoft, and you have an amazing ecosystem to support better analytics. Heck, even Excel now supports massive data sets and 1 million-plus rows.

On the back end, a crack team of great data scientists and engineers may be happy to completely redesign your architecture. But is that your top priority today? Or would you rather take advantage of existing tools that don't need a revamp of your entire data model and spend that cash elsewhere?

Look, a $750 million-revenue organization already distributes thousands of reports and dashboards every week. People up and down the ranks make data-driven decisions all the time. That's the good news. The bad news is that most of those decisions are based on legacy practices and methodologies that don't fully leverage existing structured data sets, let alone unstructured data. So back away from the Hadoop cluster and focus on how you can improve your company's analytics skill sets and results today. It's completely doable.

Moreover, you can make your internal data work for your not only your employees, but for customers and suppliers. I work with a US software publisher that's a great example of leveraging data that would once have sat idle. The company offers reading and literacy software for grades K-12. Its system provides detailed analysis on student development over the course of the year. The current reporting is typical in education -- progress by student, grade, and school, aligned to state standards.

The publisher realized it could create an anonymous pool of performance data that would link all student progress across its entire client base. This data could be combined with socioeconomic and organizational data to create a rich "best practices" data pool that simply doesn't exist today. The data set, scheduled to go live this fall, will let a school analyze, in near real time, how students are performing relative to their class, district, and similar institutions across the United States. They can analyze specific strategies and exercises that are part of the program at an individual district level. And the best part? Reporting is built right into existing data tools, so people can tap into this treasure trove on a continual basis.

Read the rest of the story in the new issue of
InformationWeek Tech Digest (free registration required).
Mike Healey is the president of Yeoman Technology Group, an engineering and research firm focusing on maximizing technology investments for organizations, and an InformationWeek contributor. He has more than 25 years of experience in technology integration and business ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
shamika
50%
50%
shamika,
User Rank: Ninja
8/31/2014 | 7:35:47 AM
Re: A tough choice for companies
 I agree with you. However it is very difficult to find people with this skill.  Data analysis can be used in any industry to understand where they stand.
shamika
50%
50%
shamika,
User Rank: Ninja
8/31/2014 | 7:24:08 AM
Re: Educattion will be key
Interesting article. As per my understanding, data analytics will be mainly used in employments. However this is not fall under any of the main categories in the education systems. Therefore it is important to add this concept in to the current education system.
yalanand
50%
50%
yalanand,
User Rank: Ninja
8/30/2014 | 2:37:49 PM
A tough choice for companies
Big Data analytics is certainly an integral part of businesses today due to its power to give companies critical insights into what they can do with their data. Proper Big Data analytics presents businesses with important risk mitigation tools, enhance business operations, enrich customer experience, and lift efficiency and ultimately increase profits.

The sheer volume, velocity and variety of Big Data is likely to overwhelm companies if they are to wait for Big Data scientists for them to act on it, and even when they find the experts, they can be too costly. Big Data analytics brings a much welcome alternative to combine, contrast and analyzetrends and other important information about a company's different data sets.
pfretty
50%
50%
pfretty,
User Rank: Moderator
8/28/2014 | 1:40:43 PM
Turning data into insights
I agree the end goal has to be to get the hands into those who need the insights rather then relying on data scientists. After all, there are far too many users needing insights for data scientists to ever keep up with the demand.  However as others have posted education is going to be key to success here.  As a recent IDG SAS poll shows only 11 percent say their organizations are extremely capable of knowing what questions to ask. However, they does not mean they do not have the interest or drive to learn and adapt. We really are just getting started. 

Peter Fretty 
Alison_Diana
50%
50%
Alison_Diana,
User Rank: Author
8/26/2014 | 3:38:21 PM
Re: Educattion will be key
I think we'll get there, in terms of better software algorithms. When you think about it, user-oriented analytics are relatively new so everyone is in learning mode. As the tools get better and developers become more adept at thinking like business users, their applications will be much simpler and intuitive, too. And then, as you say, data scientists will focus on really challenging questions and areas. 
SachinEE
50%
50%
SachinEE,
User Rank: Ninja
8/26/2014 | 3:34:12 PM
Re: Educattion will be key
If universities could call companies to make data scientists visible, then that would be great. A lot of talented scientists/engineers find less exposure. They may not be from a fancy university, but their knowledge is what matters the most. Universities have to be more upfront about students working with IT as well. 
SachinEE
50%
50%
SachinEE,
User Rank: Ninja
8/26/2014 | 3:28:16 PM
Re: Educattion will be key
Or better still, hiring better software architects and then making algorithms based on the works of those data scientists to give the end user a more streamlined version of analytics would be a better idea altogether. Data scientists are few and they should be put to better use.
Alison_Diana
50%
50%
Alison_Diana,
User Rank: Author
8/26/2014 | 9:28:10 AM
Re: Educattion will be key
In speaking to big data and analytics software developers, the unanimous goal appears to be to deliver analytics capabilities to end users -- while also offering advanced services via data scientists. Just as most people now can create a website, tweak photos, or make a video, we still have specialists whose work is far superior and who we hire for special projects or professional needs. I view big data analytics in a similar way; if I know the questions -- which demographic uses my product; what are the top support questions; who has the highest or lowest sales -- then simple tools are accessible to many non-science employees. However, when you don't know the questions because they incorporate multiple tiers of data, that's time to call in the big guns with their PhDs and years of experience pulling together disparate sources to discern new patterns.
jries921
100%
0%
jries921,
User Rank: Ninja
8/25/2014 | 3:22:33 PM
Educattion will be key
I see a large port of the role of PhD statiticians as teaching laypeople how to analyze data, and understand what they're looking at (what my boss taught a rookie programmer 20 or so years ago).  Data analysts don't necessarily need math or computer science degrees, though they certainly help; but they do need to have some understanding of what they're doing and why.  Universities *can* do that, but probably won't (or rather, students won't take the classes) unless employers pay them to do so; software developers can make the job easier, but software will happily churn out bad models without some informed human guidance; rather, employers are going to need to take the lead on training people, which is what they should have been doing all along.

 
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
8/25/2014 | 3:01:06 PM
Best practice idea from enterprise applications vendor
The idea this educational came up with seems like a clone of ERP market best practices where by the metrics and financials of customers of SAP, for example, are aggregated and turned into by-industry benchmarks for all SAP customers. Workday and other ERP vendors have also come up with these types of metrics. It's not really a big data thing... and metrics aren't necessarily analytical or predictive in nature. There are probably reported stats that serve as a point of comparison for other reported stats.

Having made these points, it does sound like a big win for the educational company, and happen to agree with the point that you can embrace big-data techniques and high-scale data without hiring a bunch of PhDs.
In A Fever For Big Data
In A Fever For Big Data
Healthcare orgs are relentlessly accumulating data, and a growing array of tools are becoming available to manage it.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Government Oct. 20, 2014
Energy and weather agencies are busting long-held barriers to analyzing big data. Can the feds now get other government agencies into the movement?
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
A roundup of the top stories and trends on InformationWeek.com
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.