Why Big Data Tools Are Here To Stay: InformationWeek Video
Nix the naysayers. Big data leaders say they need new tools, and they're not letting talent shortages get in the way.
Media coverage of big data tends to fall into two broad categories: stories that are abstract, philosophical, or speculative about what big data is all about and how it will or won't change the world; and more-concrete analyses about specific new capabilities or actual projects delivering results.
InformationWeek tries to stick to the latter, so we invited three high-level executives leading successful big data initiatives to our InformationWeek Conference, held in Las Vegas in April. We've written about all three of these projects in detail, but meeting these leaders in person gave us a chance to do one-on-one video interviews with them, in which each executive had definitive things to say about the need for new big-data technologies.
Bryson Koehler, CIO of The Weather Company (parent of The Weather Channel), is moving the company into the cloud while at the same time building a platform to capture and make use of 20 terabytes of weather-sensor data per day. That project is yielding more accurate predictions while also delivering weather data as a service worldwide.
About two minutes into the video below, Koehler explains why he's using a globally distributed NoSQL database, Riak, to underpin that new platform.
George Llado, VP of IT at Merck & Co., led a project to analyze 10 years of manufacturing data on Hadoop to optimize the production of a cancer-fighting vaccine. This wasn't a case of extreme scale -- it involved only 1.5 terabytes of data -- but in the video below, Llado explains that the variety of the data (16 disparate data sources) and the intensity of analysis (5.5 million batch comparisons and 15 billion calculations) demanded the use of Hadoop rather than conventional relational tools.
Among those abstract stories about big data I alluded to in the first paragraph, we're starting to see cautionary tales with headlines such as "Why analyzing big data can be bad for business" and "Beware 'Big Data' Hype." The hype does need to be deflated, and there's always room for caution, but a favorite theme in these big-data-backlash stories is how complicated these new platforms are and how you'll need to hire hard-to-find, expensive data scientists.
Andrew Robbins, CEO of digital marketing firm Paytronix, paid no attention to that observation. Paytronix has just 65 employees and a data management staff of 10. As he describes toward the end of the video below, he sees data science as a function, not a person. Paytronix relied on its veteran software architect, developers, and data warehousing people to correlate point-of-sale data, loyalty program data, and social data on Hadoop. Paytronix did hire a few new data-savvy people right out of college, but Robbins didn't delay the project or break the budget by waiting to find data scientists commanding six-figure salaries.
No doubt we're going to see more big-data-backlash stories. But before you let them burst your bubble, look to these real-world examples and remember that there are new data types to be exploited, and new applications that demand unprecedented scale. New opportunities and requirements have ushered in new tools, such as Hadoop and NoSQL. Yes, these tools need to evolve and mature for enterprise use, but they're not going away.
Don't fear new technologies. Trust your people to learn and evolve, and get your hands a little dirty before you try to hire your way into the future.
You can use distributed databases without putting your company's crown jewels at risk. Here's how. Also in the Data Scatter issue of InformationWeek: A wild-card team member with a different skill set can help provide an outside perspective that might turn big data into business innovation. (Free registration required.)
Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of ... View Full Bio
6 Tools to Protect Big DataMost IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Big Data Brings Big Security ProblemsWhy should big data be more difficult to secure? In a word, variety. But the business won’t wait to use it to predict customer behavior, find correlations across disparate data sources, predict fraud or financial risk, and more.
InformationWeek Tech Digest, Nov. 10, 2014Just 30% of respondents to our new survey say their companies are very or extremely effective at identifying critical data and analyzing it to make decisions, down from 42% in 2013. What gives?