Hadoop Summit: Momentum, Momentum, Momentum - InformationWeek
Data Management // Big Data Analytics
12:20 PM
Ransomware: Latest Developments & How to Defend Against Them
Nov 01, 2017
Ransomware is one of the fastest growing types of malware, and new breeds that escalate quickly ar ...Read More>>

Hadoop Summit: Momentum, Momentum, Momentum

At Hadoop Summit this week, Hortonworks will point to increasing adoption and Hadoop's shift from trendy to mainstream technology.

10 Big Data Pros To Follow On Twitter
10 Big Data Pros To Follow On Twitter
(Click image for larger view and slideshow.)

Perhaps Steve Ballmer's available to whip the crowd at this week's Hadoop Summit into a frenzy with an encore of his timeless "developers, developers, developers" performance, only chanting the word "momentum" instead.

Expect momentum -- lots of it -- to be a pervasive idea running throughout the three-day event, which kicks off Tuesday in San Jose, Calif., and will be streamed through a free online broadcast. From jobseekers to vendors to corporate users just dipping their toes in the big-data water, Hadoop's got the forward momentum of an Olympic bobsled team.

"The pace of Hadoop usage is certainly accelerating," Hortonworks VP of marketing Dave McJannet said in an interview. "This emergence of a common data architecture with Hadoop as a core component has landed."

[Big data leaders say they need new tools, and they're not letting talent shortages get in the way. See Why Big Data Tools Are Here To Stay: InformationWeek Video.]

Hortonworks, which organizes the Hadoop Summit alongside Yahoo, expects more than 3,000 participants at this week's event, as well as more than 80 sponsor organizations, up from around 60 at the previous summit. McJannet points to the release of YARN last fall as one of the major drivers of increasing adoption and a shift from trendy technology to mainstream technology. YARN enables more dynamic resource management within clusters beyond batch-oriented MapReduce jobs, so that organizations can run multiple applications in the same cluster. Perhaps, McJannet gives as an example, in addition to that MapReduce job you also want to do interactive query and stream processing on that data set as well.

"YARN, more than anything, [moved] these single-application clusters, which they may have been running in the [Hadoop 1.0] world, to these multi-application clusters where all of sudden they'll have five, six, seven, eight" applications running on a single cluster, says McJannet. "That's leading many more end-users to start using it, and leading more large IT [vendors] to start engaging with it."

As a result, you can likewise expect YARN to be mentioned quite a bit in tandem with that momentum. Hortonworks, for one, announced on Tuesday a "YARN Ready Program" as part of its partner certification program. The name's pretty self-explanatory: The program aims to help integrated software vendors (ISVs) and other partners get new users and their existing applications into Hadoop without tons of painful retrofitting required. This coincides with the tech preview of Apache Slider, a technology for porting applications into YARN and Hadoop without having to rewrite them from scratch.

Back to that momentum: As Hadoop continues to evolve into a platform, not just for tech companies and big-data startups, but for organizations in almost any industry, the next waves of corporate adopters will pose three key questions before making the leap, according to McJannet. First: Why Hadoop? Second: Does this integrate with the technologies I already use? Third: How do I leverage my existing IT skills?

On the "Why Hadoop?" front, McJannet noted a common path for new adoption: building a single analytics application around a particular data set, such as clickstream, server log, or machine data. Then, the organization builds additional applications from there. YARN's ability to support multiple data processing engines in a single cluster was a boon for such piece-by-piece projects, which can be more efficient and palatable for some organizations -- and less ominous than some vague "Let's do big data, now" edict handed down from the C suite.

Existing skills and legacy systems go hand-in-hand -- companies are less keen on taking the Hadoop plunge if their standing infrastructure, applications, and human resources go to waste. That's why you'll see more hardware pre-engineered to work with Hadoop, as well as partnerships between vendors like Hortonworks and IT stalwarts like Microsoft, Teradata, SAS, SAP, HP, Red Hat, and others.

"If you have a Teradata investment, if you have an Excel investment, if you have an SQL server investment, you can plug in Hadoop as a component of that architecture so people can use their existing skills," McJannet told us. Again, YARN's a driving force: You no longer need developers versed in MapReduce to use Hadoop. "As mainstream adopters start [using] Hadoop en masse, the ability to leverage skills and integrate with existing technologies are probably number one and number two in their decision criteria."

OK, so you probably won't see Ballmer dance across the Hadoop Summit stage -- he's keeping busy in retirement -- but the mantra stands: Momentum, Momentum, Momentum.

Can the trendy tech strategy of DevOps really bring peace between developers and IT operations -- and deliver faster, more reliable app creation and delivery? Also in the DevOps Challenge issue of InformationWeek: Execs charting digital business strategies can't afford to take Internet connectivity for granted.

Kevin Casey is a writer based in North Carolina who writes about technology for small and mid-size businesses. View Full Bio

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Ninja
6/9/2014 | 3:23:19 AM
Widespread exposure brings nasty surprises
From trendy to mainstream brings more exposure to a service/product that would have previously gone unnoticed, but the limelight doesn't always bring good things.  Case in point:  Ubuntu Linux. The only problem with moving to mainstream was that user privacy was the pricetag.  In Ubuntu's case, the CEO of Canonical (Ubuntu's Linux distributor) is a former government contractor previously tasked with writing spyware. And now Ubuntu comes with third party snooping on internet searches conducted by the user. Makes me wonder what nasty surprise(s) is/are in store for users now that Hadoop has also been adopted by the mainstream.
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of IT Report
In today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll