IoT
IoT
Data Management // Software Platforms
Commentary
12/8/2014
11:30 AM
Doug Henschen
Doug Henschen
Commentary
Connect Directly
Google+
LinkedIn
Twitter
RSS
100%
0%

Top 5 Big Data Trends Of 2014

As companies move beyond bleeding-edge experiments into production deployments, these trends point to real-world progress in big data analysis.

InformationWeek Chiefs Of The Year: Where Are They Now?
InformationWeek Chiefs Of The Year: Where Are They Now?
(Click image for larger view and slideshow.)

The era of big data analysis is here to stay. Take your pick of 2014 proof points.

Tech watchers might cite the more than $200 million in venture capital raised by the top three NoSQL database vendors, or the $1 billion raised by the top-three Hadoop software distributors. Many took note of the recent declaration by Forrester Research that "Hadoop is no longer optional" for large enterprises, thanks to compelling "Hadooponomics" that make it a must for high-scale storage and data processing.

InformationWeek is more impressed by the testimonials of companies that are getting real value out of big data platforms and analysis techniques. Pfizer and Merck, for example, are developing more effective and affordable drugs thanks to big data techniques that are leading to more targeted treatments and more productive manufacturing processes. GE and others are demonstrating improvements in industrial equipment performance, uptime, and safety thanks to Internet of things-style applications.

[Want more on the top IT achiever of 2014? Read IT Chief Of The Year: Bank Of America's Cathy Bessant.]

And then there are the pioneers like The Weather Company and Facebook that say they just couldn't run their data-driven businesses without new platforms, even if they still have a place for more conventional tools like relational databases.

Here are five trends witnessed over the last year that point to progress in big data analysis:

1. SQL meets Hadoop
Hadoop is here to stay, so every data management vendor worth its salt must have a SQL-on-Hadoop or SQL-access-to-Hadoop option. Here are five of our most-read stories in the SQL-meets-Hadoop vein:

Just remember that SQL is not designed to find correlations among variably structured data sets. Nor does it support machine learning, many advanced analytics techniques, or other approaches often associated with big data analysis. If SQL solved everything, we wouldn't need new platforms.

2. Platforms mature
Every other week in 2014, or so it seems, Hadoop software distributors and NoSQL database vendors announced new management consoles, security systems, data management capabilities, search engines, or high-availability features. Here's a sampling of what we're talking about:

These and other big data vendors are trying to reassure enterprise IT types that these products are secure and reliable as 30-year-old database management systems. Let's just say that more than a few grizzled IT veterans are still used to working with favored and familiar tools and still need some convincing.

3. Educational options proliferate
Nature abhors a vacuum, so into the void of data science and big-data platform knowledge and expertise have rushed vendors,

Next Page

Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of ... View Full Bio
Previous
1 of 2
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 2   >   >>
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
12/15/2014 | 11:31:57 AM
Re: SQL on Hadoop
HAWQ was introduced in 2013, not 2014, so it was ahead of the 2014 rush of followers. I haven't seen any difinitive benchmarks about HAWQ versus any other SQL-on-Hadoop option ("Stinger" or otherwise). I've also seen no evidence of big HAWQ adoption. If it's so great, why haven't we seen lots of HAWQ users sharing their stories? I'm open to writing about them if Pivotal can make customers available. Ditto Presto, which I hear about mainly from Presto advocates (not practitioners).
pfretty
50%
50%
pfretty,
User Rank: Ninja
12/15/2014 | 11:23:23 AM
Love the list
I think you are spot on with your trend identification.  I would like to see a little more focus on developing, implementing and nurturing strategies. According to a recent IDG survey, data strategy development is still lagging considerably, which could prove discouraging to organizations not realizing the ROI as quickly as they hope. 


Peter Fretty, j.mp/pfrettysa
tspann085
50%
50%
tspann085,
User Rank: Apprentice
12/15/2014 | 10:08:19 AM
SQL on Hadoop
You cannot forget about the fastest SQL engine on Hadoop, Pivotal HAWQ which is much faster than Stringer or Presto.    It also has full SQL support and can run all TPC-DS 111 queries.
HM
50%
50%
HM,
User Rank: Moderator
12/9/2014 | 3:43:34 PM
Big Data Alternative
Doug, interesting trends on Big Data. Hadoop alternatives such as HPCC Systems provides proven solutions to handle what are now called Big Data problems, and have been doing so for more than a decade. More info at http://hpccsystems.com.

 
gkumar_splice
50%
50%
gkumar_splice,
User Rank: Apprentice
12/9/2014 | 3:09:54 PM
Re: A remaining challenge?
I agree with Charlie's comment, and I believe in addition to real-time, we need to also ensure that operational applications, which require high concurrency and consistency requirements, can leverage Hadoop's scalability and fault-tolerant capabilities. Splice Machine's RDBMS offers organizations these capabilities on top of the Hadoop stack - real-time updates, support for applications with 100s of users and high concurrency, and those which require consistency guarantees. 
Laurianne
50%
50%
Laurianne,
User Rank: Author
12/9/2014 | 9:34:10 AM
2015
Will 2015 be the year more companies build/train their own data analysis gurus? I hope we continue to see more IT people who may not have formal analytics training establish their expertise via project work.
yamtssfa
50%
50%
yamtssfa,
User Rank: Apprentice
12/9/2014 | 6:43:29 AM
Big data - but from where?
I think it's also important to mention the trends of data acquisition, either private data (the data service companies collect about us), or aggregated data (Gnip/Datasift), or publicly available data (Webhose.io/Moreover)
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
12/8/2014 | 7:04:59 PM
Re: A remaining challenge?
It would be fair to say that streaming analysis remains a challenge, but there are multiple initiatives trying to address that problem. In the big data arena examples include Spark, Storm, Amazon Kinesis. NoSQL and NewSQL vendors and options are also sprouting analytical features, but I'd say this trend hasn't really hit the production mainstream this year.
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
12/8/2014 | 6:56:42 PM
A remaining challenge?
Would it be fair to say that a remaining challenge for big data practitioners is getting their systems to produce results closer to real time, as opposed to batch timje? So much could be done if we could instantly extract the nuggets we need from the big data generated on active Web sites or in ecommerce applications. Getting useful data to respond to a site visitor, or to recognize a significant event amidst a stream of them -- that's a big challenge.
Stratustician
50%
50%
Stratustician,
User Rank: Ninja
12/8/2014 | 3:33:04 PM
is 2015 the year of big data?
Nice to see some actual use cases for Hadoop start to come to light.  A year ago folks were still trying to figure out it seems whether Big Data analytics was more than hype.  I guess here we have some great proof points as to why it's going to be critical for many organizations moving forward.  Even nicer to see is that the right educational tools are in place to support these new areas of IT.
Page 1 / 2   >   >>
In A Fever For Big Data
In A Fever For Big Data
Healthcare orgs are relentlessly accumulating data, and a growing array of tools are becoming available to manage it.
Register for InformationWeek Newsletters
White Papers
Current Issue
Top IT Trends to Watch in Financial Services
IT pros at banks, investment houses, insurance companies, and other financial services organizations are focused on a range of issues, from peer-to-peer lending to cybersecurity to performance, agility, and compliance. It all matters.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.