Why Big Data Tools Are Here To Stay: InformationWeek Video - InformationWeek
Data Management // Big Data Analytics
10:20 AM
Doug Henschen
Doug Henschen
Connect Directly

Why Big Data Tools Are Here To Stay: InformationWeek Video

Nix the naysayers. Big data leaders say they need new tools, and they're not letting talent shortages get in the way.

Media coverage of big data tends to fall into two broad categories: stories that are abstract, philosophical, or speculative about what big data is all about and how it will or won't change the world; and more-concrete analyses about specific new capabilities or actual projects delivering results.

InformationWeek tries to stick to the latter, so we invited three high-level executives leading successful big data initiatives to our InformationWeek Conference, held in Las Vegas in April. We've written about all three of these projects in detail, but meeting these leaders in person gave us a chance to do one-on-one video interviews with them, in which each executive had definitive things to say about the need for new big-data technologies.

Bryson Koehler, CIO of The Weather Company (parent of The Weather Channel), is moving the company into the cloud while at the same time building a platform to capture and make use of 20 terabytes of weather-sensor data per day. That project is yielding more accurate predictions while also delivering weather data as a service worldwide.

About two minutes into the video below, Koehler explains why he's using a globally distributed NoSQL database, Riak, to underpin that new platform.


George Llado, VP of IT at Merck & Co., led a project to analyze 10 years of manufacturing data on Hadoop to optimize the production of a cancer-fighting vaccine. This wasn't a case of extreme scale -- it involved only 1.5 terabytes of data -- but in the video below, Llado explains that the variety of the data (16 disparate data sources) and the intensity of analysis (5.5 million batch comparisons and 15 billion calculations) demanded the use of Hadoop rather than conventional relational tools.


Among those abstract stories about big data I alluded to in the first paragraph, we're starting to see cautionary tales with headlines such as "Why analyzing big data can be bad for business" and "Beware 'Big Data' Hype." The hype does need to be deflated, and there's always room for caution, but a favorite theme in these big-data-backlash stories is how complicated these new platforms are and how you'll need to hire hard-to-find, expensive data scientists.

Andrew Robbins, CEO of digital marketing firm Paytronix, paid no attention to that observation. Paytronix has just 65 employees and a data management staff of 10. As he describes toward the end of the video below, he sees data science as a function, not a person. Paytronix relied on its veteran software architect, developers, and data warehousing people to correlate point-of-sale data, loyalty program data, and social data on Hadoop. Paytronix did hire a few new data-savvy people right out of college, but Robbins didn't delay the project or break the budget by waiting to find data scientists commanding six-figure salaries.


No doubt we're going to see more big-data-backlash stories. But before you let them burst your bubble, look to these real-world examples and remember that there are new data types to be exploited, and new applications that demand unprecedented scale. New opportunities and requirements have ushered in new tools, such as Hadoop and NoSQL. Yes, these tools need to evolve and mature for enterprise use, but they're not going away.

Don't fear new technologies. Trust your people to learn and evolve, and get your hands a little dirty before you try to hire your way into the future.

You can use distributed databases without putting your company's crown jewels at risk. Here's how. Also in the Data Scatter issue of InformationWeek: A wild-card team member with a different skill set can help provide an outside perspective that might turn big data into business innovation. (Free registration required.)

Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Moderator
6/3/2014 | 9:40:24 AM
HPCC Systems
Doug, very informative article. We are seeing an increase in businesses seeking specialized skills to help address challenges that arose with the era of big data. The HPCC Systems platform from LexisNexis helps to fill this gap by allowing data analysts themselves to own the complete data lifecycle. Designed by data scientists, ECL is a declarative programming language used to express data algorithms across the entire HPCC platform. Their built-in analytics libraries for Machine Learning and BI integration provide a complete integrated solution from data ingestion and data processing to data delivery. HPCC Systems provides proven solutions to handle what are now called Big Data problems, and have been doing so for more than a decade. More at http://hpccsystems.com
User Rank: Author
6/2/2014 | 7:07:44 PM
Re: Big data hiring
Merck is a fascinating example -- not a huge quantity of data, but complex in terms of variety. I thought it very interesting that Merck started thinking about Hadoop and cloud infrastructure, and found out that its research department had experience with that.
User Rank: Author
6/2/2014 | 11:08:13 AM
Big data hiring
The Paytronix hiring advice mirrors what I am hearing from more people in the big data community now: Training internally is not only possible abut also necessary. Or as The Weather Channel CIO remarked at the InformationWeek conference, hire for curiousity.
The Staying Power of Legacy Systems
Mary E. Shacklett, Mary E. Shacklett,  4/15/2019
Q&A: Red Hat's Robert Kratky Discusses Essentials of Docs
Joao-Pierre S. Ruth, Senior Writer,  4/15/2019
How Cloud Shifts Security Balance of Power to the Good Guys
Guest Commentary, Guest Commentary,  4/11/2019
White Papers
Register for InformationWeek Newsletters
Current Issue
A New World of IT Management in 2019
This IT Trend Report highlights how several years of developments in technology and business strategies have led to a subsequent wave of changes in the role of an IT organization, how CIOs and other IT leaders approach management, in addition to the jobs of many IT professionals up and down the org chart.
Flash Poll