Why Big Data Tools Are Here To Stay: InformationWeek Video - InformationWeek
Data Management // Big Data Analytics
10:20 AM
Doug Henschen
Doug Henschen
Connect Directly

Why Big Data Tools Are Here To Stay: InformationWeek Video

Nix the naysayers. Big data leaders say they need new tools, and they're not letting talent shortages get in the way.

Media coverage of big data tends to fall into two broad categories: stories that are abstract, philosophical, or speculative about what big data is all about and how it will or won't change the world; and more-concrete analyses about specific new capabilities or actual projects delivering results.

InformationWeek tries to stick to the latter, so we invited three high-level executives leading successful big data initiatives to our InformationWeek Conference, held in Las Vegas in April. We've written about all three of these projects in detail, but meeting these leaders in person gave us a chance to do one-on-one video interviews with them, in which each executive had definitive things to say about the need for new big-data technologies.

Bryson Koehler, CIO of The Weather Company (parent of The Weather Channel), is moving the company into the cloud while at the same time building a platform to capture and make use of 20 terabytes of weather-sensor data per day. That project is yielding more accurate predictions while also delivering weather data as a service worldwide.

About two minutes into the video below, Koehler explains why he's using a globally distributed NoSQL database, Riak, to underpin that new platform.

George Llado, VP of IT at Merck & Co., led a project to analyze 10 years of manufacturing data on Hadoop to optimize the production of a cancer-fighting vaccine. This wasn't a case of extreme scale -- it involved only 1.5 terabytes of data -- but in the video below, Llado explains that the variety of the data (16 disparate data sources) and the intensity of analysis (5.5 million batch comparisons and 15 billion calculations) demanded the use of Hadoop rather than conventional relational tools.

Among those abstract stories about big data I alluded to in the first paragraph, we're starting to see cautionary tales with headlines such as "Why analyzing big data can be bad for business" and "Beware 'Big Data' Hype." The hype does need to be deflated, and there's always room for caution, but a favorite theme in these big-data-backlash stories is how complicated these new platforms are and how you'll need to hire hard-to-find, expensive data scientists.

Andrew Robbins, CEO of digital marketing firm Paytronix, paid no attention to that observation. Paytronix has just 65 employees and a data management staff of 10. As he describes toward the end of the video below, he sees data science as a function, not a person. Paytronix relied on its veteran software architect, developers, and data warehousing people to correlate point-of-sale data, loyalty program data, and social data on Hadoop. Paytronix did hire a few new data-savvy people right out of college, but Robbins didn't delay the project or break the budget by waiting to find data scientists commanding six-figure salaries.

No doubt we're going to see more big-data-backlash stories. But before you let them burst your bubble, look to these real-world examples and remember that there are new data types to be exploited, and new applications that demand unprecedented scale. New opportunities and requirements have ushered in new tools, such as Hadoop and NoSQL. Yes, these tools need to evolve and mature for enterprise use, but they're not going away.

Don't fear new technologies. Trust your people to learn and evolve, and get your hands a little dirty before you try to hire your way into the future.

You can use distributed databases without putting your company's crown jewels at risk. Here's how. Also in the Data Scatter issue of InformationWeek: A wild-card team member with a different skill set can help provide an outside perspective that might turn big data into business innovation. (Free registration required.)

Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of ... View Full Bio
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Moderator
6/3/2014 | 9:40:24 AM
HPCC Systems
Doug, very informative article. We are seeing an increase in businesses seeking specialized skills to help address challenges that arose with the era of big data. The HPCC Systems platform from LexisNexis helps to fill this gap by allowing data analysts themselves to own the complete data lifecycle. Designed by data scientists, ECL is a declarative programming language used to express data algorithms across the entire HPCC platform. Their built-in analytics libraries for Machine Learning and BI integration provide a complete integrated solution from data ingestion and data processing to data delivery. HPCC Systems provides proven solutions to handle what are now called Big Data problems, and have been doing so for more than a decade. More at http://hpccsystems.com
User Rank: Author
6/2/2014 | 7:07:44 PM
Re: Big data hiring
Merck is a fascinating example -- not a huge quantity of data, but complex in terms of variety. I thought it very interesting that Merck started thinking about Hadoop and cloud infrastructure, and found out that its research department had experience with that.
User Rank: Author
6/2/2014 | 11:08:13 AM
Big data hiring
The Paytronix hiring advice mirrors what I am hearing from more people in the big data community now: Training internally is not only possible abut also necessary. Or as The Weather Channel CIO remarked at the InformationWeek conference, hire for curiousity.
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of the Cloud Report
As the use of public cloud becomes a given, IT leaders must navigate the transition and advocate for management tools or architectures that allow them to realize the benefits they seek. Download this report to explore the issues and how to best leverage the cloud moving forward.
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of November 6, 2016. We'll be talking with the InformationWeek.com editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll