IoT
IoT
Data Management // Software Platforms
News
11/4/2014
09:06 AM
Connect Directly
Google+
LinkedIn
Twitter
RSS
E-Mail
50%
50%

Hadoop 'No Longer Optional,' Says Forrester

Forrester says economics will make Hadoop "mandatory," predicts new roles, new software sources, and an end to skills shortage by 2015.

HP's 3D Future: Sprout Visual Tour
HP's 3D Future: Sprout Visual Tour
(Click image for larger view and slideshow.)

"Hadooponomics" will make Apache's open-source big data platform a "must have for large enterprises."

This is the first of eight predictions Forrester Research published on Tuesday in "Predictions 2015: Hadoop Will Become A Cornerstone Of Your Business Technology Agenda." Hadooponomics is "the ability to linearly scale both data storage and data processing," the report explains, and it is closely tied to the ability to leverage "pay-per-use cloudonomics."

Many enterprises are just dabbling in Hadoop, the report acknowledges, but leaders including Wal-Mart, Fidelity Investments, Sears, Verizon, USAA, Cardinal Health, Wells Fargo, Proctor & Gamble, Cablevision, Nasdaq, AutoTrader, Netflix, and Yelp have proven, mission-critical Hadoop deployments.

"The remaining minority of dazed and confused CIOs will make Hadoop a priority for 2015," the report states. "Application development and delivery professionals should be ready and waiting with a compelling use case to get started."

[Want more on this big data platform? Read Hadoop: 5 Undeniable Truths.]

The report, which was written by Mike Gualtieri along with half a dozen other Forrester analysts, offers eight predictions about Hadoop for 2015, with the most notable addressing SQL, cloud deployment, roles beyond analytics, and new sources of Hadoop software.

SQL becomes Hadoop's killer app
Fast, ANSI-compliant SQL-on-Hadoop options will "create immediate opportunities for Hadoop to become a useful data platform for enterprises" because they will be familiar to data-management professionals and accessible to existing systems, says Forrester. This will provide a sandbox for analysis of data that is not currently accessible.

Hadoop clusters bloom in the cloud
Storage, compute, and networking capacity needs are rarely in sync, so cloud elasticity will be a key to cost efficiency, the report argues. Thus, adopters will increasingly use Hadoop in the cloud to meet demand for ad hoc analytics, yet on-premises adoption "won't slow down."

Hadoop won't be just for analytics
With Hadoop's YARN management layer, options including HBase, the transactional NoSQL database, and the Apache Spark in-memory platform will turn Hadoop into an application platform that can run code in combination with data. This will "collapse the application stack," and vendors that offer middleware, databases, search engines, integration tools, and other software infrastructure "must move to develop versions that will run natively inside Hadoop," Forrester warns.

New Hadoop sources emerge
Not only does Forrester foresee new Hadoop distributions emerging from the likes of HP, Oracle, SAP, Software AG, and Tibco, it "sees no reason" why Microsoft, Red Hat, VMware, and other operating system vendors can't include Hadoop and make it a configurable option from within their operating systems. "This would be a game-changer because it would mean that every node could be configured as a Hadoop node," Gualtieri writes. "This would disrupt the existing model used by Hadoop distribution vendors that charge $2,000 to $3,000 per node, per year for their current distributions."

Perhaps Forrester's most surprising prediction is that the Hadoop skills shortage will disappear in 2015. Understanding a distributed file system and distributing computing platform is "no big deal," Forrester says, and the Java APIs are "nothing new" to Java application developers.

"CIOs won't have to hire high-priced Hadoop consultants to get projects done," according to the report. "Hadoop projects will get done faster because the enterprise's very own application developers and operations professionals know the data, the integration points, the applications, and the business challenges."

This last prediction seems particularly optimistic, as the collective understanding of Hadoop is not likely to turn on like a light bulb in 2015. What's more, we're guessing that Hadoop won't be perceived as much of a "killer" platform if SQL is the pinnacle of its analysis capabilities. Machine learning, high-scale correlation, streaming analysis, and the discovery of important, predictive patterns across new and old data types is the stuff that will deliver real breakthroughs.

What will you use for your big data platform? A high-scale relational database? NoSQL database? Hadoop? Event-processing technology? One size doesn't fit all. Here's how to decide. Get the new Pick Your Platform For Big Data issue of InformationWeek Tech Digest today. (Free registration required.)

Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
RajaT011
50%
50%
RajaT011,
User Rank: Apprentice
4/17/2015 | 3:22:07 AM
Re: Too much focus on Hadoop?
It's the new de facto standard for data storage and processing. There could be other data platforms that are purpose built and much better than Hadoop platform, but it's the adoption and open source that's driving Hadoop.

With introduction of YARN framework, it just opened gates for new ways of processing data. Literally, there are no limits on type of data and size of data that we can store in Hadoop and data processing methods.

In the next year or so, we will see many applications are built for Hadoop. It's inevitable. We are already seeing this shift in the industry very fast.

It's the new standard for DATA. If organizations are not looking at Hadoop I think they are falling behind. It's only my perspective.
TD@ETR
50%
50%
TD@ETR,
User Rank: Apprentice
11/20/2014 | 9:25:58 AM
Re: Too much focus on Hadoop?
This is very real.  We began publishing on this as soon as we concluded our 2H14 Technology Spending Intentions Survey in early August (Hadoop Emerges in Earnest) and then again in September with our Hadoop Benefactor List. We have caught significant adoptions and increases in spend for the 7 major Hadoop vendors and at the direct detriment of the traditional EDW and ETL players.  This is most noticebale among one of the widely recognized early adopter verticals; Financials/Insurance organiizations.

 
sshaik56001
50%
50%
sshaik56001,
User Rank: Apprentice
11/11/2014 | 11:56:11 PM
Re: Too much focus on Hadoop?
In line with the Hadoop heat map cited above, which skill sets does a business analyts should familiarze/get hold with ? 
marksmithusa
50%
50%
marksmithusa,
User Rank: Apprentice
11/5/2014 | 3:54:33 PM
Re: Too much focus on Hadoop?
No, it isn't. Hadoop is very "1980s-esque" to me and it's not remotely close to as prevalent as this claims. 
danielcawrey
50%
50%
danielcawrey,
User Rank: Ninja
11/4/2014 | 11:50:36 PM
Re: Too much focus on Hadoop?
I agree with this assessment; Hadopp is going to become a necessity in enterprise environments. Because of it becoming standardized, that's going to mean less reliance on consultants and more on internal resources. 

Consultants come in when something is new. When a technology becomes an essential part of IT, the game changes. 
rcasey
IW Pick
100%
0%
rcasey,
User Rank: Strategist
11/4/2014 | 3:04:23 PM
Re: Too much focus on Hadoop?
I would say that it *is* part of a generation of technologies fighting for mindshare, but for Hadoop, it has some advantages that make it stand out.
  1. Hadoop has a notable pedigree, having its origins with the MapReduce and distributed storage efforts of Yahoo and Google.
  2. Hadoop has gained a level of ubiquity that has turned out a large amount of talent familiar with the technology, so you have a ready base of talent to hire to get the results you need.
  3. Importantly for a lot of organizations:  Hadoop can be found in Open Source.  This not only makes it freely available but gives organizations the capability of better understanding its capabilities and allowing that organization the ability full control of derivative versions of the implementation.

There are lots of ways that distributed storage, map-reduce, and resource management clusters can be implemented (and have been over the years), but many suffer from vendor lock-in, expense, opacity requiring expensive training, or simply not being an appropriate fit to the specific problem.  Hadoop would seem to offer a substrate that gets past all of these blockers and opens the door for easy to manage, relatively inexpensive high capacity clusters.
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
11/4/2014 | 1:03:27 PM
Re: Too much focus on Hadoop?
David, I wouldn't sweat it. If you weren't covering data warehousing, data marts, business intelligence and analytics before this era, you probably won't be spending much time on data lakes/data hubs etc. going forward. The bottom line for business will be data-informed predictions and prescriptive recommendations. Hopefully these can be exposed as services so those building and consuming apps won't have to worry about the data-intensive work going on behind the scenes.
David F. Carr
50%
50%
David F. Carr,
User Rank: Author
11/4/2014 | 10:37:47 AM
Re: Too much focus on Hadoop?
Data lake? I must be falling behind on my tech jargon.
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
11/4/2014 | 10:19:16 AM
Re: Too much focus on Hadoop?
No, it's really the case that there's nothing on an equal footing with Hadoop in the pantheon of emerging big-data technologies. There are a variety of NoSQL databases for different use cases and with different degrees of adoption. But only Hadoop is being widely adopted and supported by vendors as the centerpiece data lake/data hub for capturing and economically retaining enterprise data. There are many tools to analyze, report on, and predict against data, but you have to have the data platform in place to drive all those possibilities. That's the foundation.
David F. Carr
100%
0%
David F. Carr,
User Rank: Author
11/4/2014 | 10:05:54 AM
Too much focus on Hadoop?
Is Hadoop really that unique? Or is it part of a generation of new data technologies that are all vying for attention? I didn't think the focus on Hadoop was so unanimous as Forrester makes it out to be.
In A Fever For Big Data
In A Fever For Big Data
Healthcare orgs are relentlessly accumulating data, and a growing array of tools are becoming available to manage it.
Register for InformationWeek Newsletters
White Papers
Current Issue
Top IT Trends to Watch in Financial Services
IT pros at banks, investment houses, insurance companies, and other financial services organizations are focused on a range of issues, from peer-to-peer lending to cybersecurity to performance, agility, and compliance. It all matters.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Listen Now InformationWeek Live for the Week of September 25, 2016
Join us for a roundup of the top stories on InformationWeek.com for the week of September 25, 2016. We'll be talking with the InformationWeek.com editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.