Hadoop 'No Longer Optional,' Says Forrester

HP's 3D Future: Sprout Visual Tour

"Hadooponomics" will make Apache's open-source big data platform a "must have for large enterprises."

This is the first of eight predictions Forrester Research published on Tuesday in "Predictions 2015: Hadoop Will Become A Cornerstone Of Your Business Technology Agenda." Hadooponomics is "the ability to linearly scale both data storage and data processing," the report explains, and it is closely tied to the ability to leverage "pay-per-use cloudonomics."

Many enterprises are just dabbling in Hadoop, the report acknowledges, but leaders including Wal-Mart, Fidelity Investments, Sears, Verizon, USAA, Cardinal Health, Wells Fargo, Proctor & Gamble, Cablevision, Nasdaq, AutoTrader, Netflix, and Yelp have proven, mission-critical Hadoop deployments.

"The remaining minority of dazed and confused CIOs will make Hadoop a priority for 2015," the report states. "Application development and delivery professionals should be ready and waiting with a compelling use case to get started."

[Want more on this big data platform? Read Hadoop: 5 Undeniable Truths.]

The report, which was written by Mike Gualtieri along with half a dozen other Forrester analysts, offers eight predictions about Hadoop for 2015, with the most notable addressing SQL, cloud deployment, roles beyond analytics, and new sources of Hadoop software.

SQL becomes Hadoop's killer app
Fast, ANSI-compliant SQL-on-Hadoop options will "create immediate opportunities for Hadoop to become a useful data platform for enterprises" because they will be familiar to data-management professionals and accessible to existing systems, says Forrester. This will provide a sandbox for analysis of data that is not currently accessible.

Hadoop clusters bloom in the cloud
Storage, compute, and networking capacity needs are rarely in sync, so cloud elasticity will be a key to cost efficiency, the report argues. Thus, adopters will increasingly use Hadoop in the cloud to meet demand for ad hoc analytics, yet on-premises adoption "won't slow down."

Hadoop won't be just for analytics
With Hadoop's YARN management layer, options including HBase, the transactional NoSQL database, and the Apache Spark in-memory platform will turn Hadoop into an application platform that can run code in combination with data. This will "collapse the application stack," and vendors that offer middleware, databases, search engines, integration tools, and other software infrastructure "must move to develop versions that will run natively inside Hadoop," Forrester warns.

New Hadoop sources emerge
Not only does Forrester foresee new Hadoop distributions emerging from the likes of HP, Oracle, SAP, Software AG, and Tibco, it "sees no reason" why Microsoft, Red Hat, VMware, and other operating system vendors can't include Hadoop and make it a configurable option from within their operating systems. "This would be a game-changer because it would mean that every node could be configured as a Hadoop node," Gualtieri writes. "This would disrupt the existing model used by Hadoop distribution vendors that charge $2,000 to $3,000 per node, per year for their current distributions."

Perhaps Forrester's most surprising prediction is that the Hadoop skills shortage will disappear in 2015. Understanding a distributed file system and distributing computing platform is "no big deal," Forrester says, and the Java APIs are "nothing new" to Java application developers.

"CIOs won't have to hire high-priced Hadoop consultants to get projects done," according to the report. "Hadoop projects will get done faster because the enterprise's very own application developers and operations professionals know the data, the integration points, the applications, and the business challenges."

This last prediction seems particularly optimistic, as the collective understanding of Hadoop is not likely to turn on like a light bulb in 2015. What's more, we're guessing that Hadoop won't be perceived as much of a "killer" platform if SQL is the pinnacle of its analysis capabilities. Machine learning, high-scale correlation, streaming analysis, and the discovery of important, predictive patterns across new and old data types is the stuff that will deliver real breakthroughs.

What will you use for your big data platform? A high-scale relational database? NoSQL database? Hadoop? Event-processing technology? One size doesn't fit all. Here's how to decide. Get the new Pick Your Platform For Big Data issue of InformationWeek Tech Digest today. (Free registration required.)