Why Big Data Tools Are Here To Stay: InformationWeek Video

Media coverage of big data tends to fall into two broad categories: stories that are abstract, philosophical, or speculative about what big data is all about and how it will or won't change the world; and more-concrete analyses about specific new capabilities or actual projects delivering results.

InformationWeek tries to stick to the latter, so we invited three high-level executives leading successful big data initiatives to our InformationWeek Conference, held in Las Vegas in April. We've written about all three of these projects in detail, but meeting these leaders in person gave us a chance to do one-on-one video interviews with them, in which each executive had definitive things to say about the need for new big-data technologies.

Bryson Koehler, CIO of The Weather Company (parent of The Weather Channel), is moving the company into the cloud while at the same time building a platform to capture and make use of 20 terabytes of weather-sensor data per day. That project is yielding more accurate predictions while also delivering weather data as a service worldwide.

About two minutes into the video below, Koehler explains why he's using a globally distributed NoSQL database, Riak, to underpin that new platform.

{videoembed|1269341}

George Llado, VP of IT at Merck & Co., led a project to analyze 10 years of manufacturing data on Hadoop to optimize the production of a cancer-fighting vaccine. This wasn't a case of extreme scale -- it involved only 1.5 terabytes of data -- but in the video below, Llado explains that the variety of the data (16 disparate data sources) and the intensity of analysis (5.5 million batch comparisons and 15 billion calculations) demanded the use of Hadoop rather than conventional relational tools.

{videoembed|1269338}

Among those abstract stories about big data I alluded to in the first paragraph, we're starting to see cautionary tales with headlines such as "Why analyzing big data can be bad for business" and "Beware 'Big Data' Hype." The hype does need to be deflated, and there's always room for caution, but a favorite theme in these big-data-backlash stories is how complicated these new platforms are and how you'll need to hire hard-to-find, expensive data scientists.

Andrew Robbins, CEO of digital marketing firm Paytronix, paid no attention to that observation. Paytronix has just 65 employees and a data management staff of 10. As he describes toward the end of the video below, he sees data science as a function, not a person. Paytronix relied on its veteran software architect, developers, and data warehousing people to correlate point-of-sale data, loyalty program data, and social data on Hadoop. Paytronix did hire a few new data-savvy people right out of college, but Robbins didn't delay the project or break the budget by waiting to find data scientists commanding six-figure salaries.

{videoembed|1269339}

No doubt we're going to see more big-data-backlash stories. But before you let them burst your bubble, look to these real-world examples and remember that there are new data types to be exploited, and new applications that demand unprecedented scale. New opportunities and requirements have ushered in new tools, such as Hadoop and NoSQL. Yes, these tools need to evolve and mature for enterprise use, but they're not going away.

Don't fear new technologies. Trust your people to learn and evolve, and get your hands a little dirty before you try to hire your way into the future.

You can use distributed databases without putting your company's crown jewels at risk. Here's how. Also in the Data Scatter issue of InformationWeek: A wild-card team member with a different skill set can help provide an outside perspective that might turn big data into business innovation. (Free registration required.)