Intel Unveils New Distribution For Apache Hadoop - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

03:52 PM
Connect Directly

Intel Unveils New Distribution For Apache Hadoop

Third generation of Intel's Hadoop promises better performance and security and tighter integration with Intel's Xeon processors.

Intel's Tech Roadmap: Visual Tour
Intel's Tech Roadmap: Visual Tour
(click image for larger view and for slideshow)
Intel Corporation today announced the availability of its latest distribution of the Apache Hadoop software. The new version, Intel's third major Hadoop release, features management tools unique to the chip giant's implementation of the open-source framework, and promises improved performance and security features.

Calling the release "a pretty significant announcement for Intel," Boyd Davis, VP of the Intel architecture group, acknowledged the company is known more for its hardware than software.

"We're announcing the delivery and a commercial offering of a software product from a company that many of you know as a chip company," Davis said at an Intel media event in San Francisco.

He quickly added, however, that Intel's involvement with Hadoop, the leading big data software framework for distributed applications, goes back a few years. "We've actually been at this for a while," he said.

[ What big tech breakthroughs are in store? Read Intel CTO: Smart Sensors, Wearable Tech Coming Soon. ]

Intel began its Hadoop efforts in 2009 when it formed a lab with Yahoo, which brought the open-source platform to market, and with HP to gain a better understanding into the nature of data-intensive workloads.

In 2011, Intel began working with some of its customers in China, including very large companies such as China Unicom and China Mobile, who were "struggling to get the value out of their Hadoop infrastructure," said Davis.

The company released its first and second major commercial releases of Hadoop in 2012. For its new release, Intel focused on performance and security enhancements. "Today (Hadoop is) often viewed as an offline, very slow, batch processing engine for relatively unsophisticated types of data," Davis said.

The new Intel distribution is the first to support the Advanced Encrytion Standard (AES) Instructions found in Intel's Xeon processor platform. Silicon-based encryption support of the Hadoop Distributed File System will allow organizations to securely analyze their data sets without hampering performance, according to Intel.

Networking and IO technology optimizations in the Xeon processor enable much faster analytic performance, the company said. For instance, analyzing one terabyte of data, which previously might have taken more than four hours, now takes about seven minutes on Intel's software and hardware platform, said Davis. "One of the things we're doing is taking advantage of our hardware platform heritage," he said. Several years ago, Intel put instructions into Xeon processors that accelerate AES, a common encryption algorithm used in disk encryption, SSL transactions online and myriad other operations.

"We collect metrics from the performance of the cluster, and then use machine-learning algorithms to suggest improvements to the configuration settings," said Davis, who added that Intel is seeing 30% performance gains with its approach.

However, the management layer in Intel's Hadoop distribution "will remain unique to Intel" and won't be shared with the open-source community. Intel's management layer isn't "intrinsic to the operation of the open source projects, so customers aren't locked into it," said Davis. "But it's a utility that makes it far easier for customers to get the most out of their infrastructure."

In the question-and-answer session that followed the announcement, Davis said that Intel doesn't necessarily see itself as a competitor to leading Hadoop providers such as Cloudera and Hortonworks, but rather as a partner that contributes to the platform's open-source community. Intel also announced dozens of Hadoop partners at the launch, including Cisco, Cray, Dell, Red Hat, SAP and Teradata.

Davis said it's important to view big data as more than simply another overhyped tech industry buzzword. "We're in an era of generating huge amounts of data," he said. "We're clearly in uncharted territory, and this doesn't even start to comprehend the kind of data that's going to be generated as we start seeing the Internet of Things, and huge arrays of sensors, cameras, and other edge devices that are connected machine to machine."

Attend Interop Las Vegas, May 6-10, and attend the most thorough training on Apple Deployment at the NEW Mac & iOS IT Conference. Use Priority Code DIPR02 by March 2 to save up to $500 off the price of Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 350+ exhibiting companies, and the latest technology. Register for Interop today!

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Enterprise Guide to Edge Computing
Cathleen Gagne, Managing Editor, InformationWeek,  10/15/2019
Rethinking IT: Tech Investments that Drive Business Growth
Jessica Davis, Senior Editor, Enterprise Apps,  10/3/2019
IT Careers: 12 Job Skills in Demand for 2020
Cynthia Harvey, Freelance Journalist, InformationWeek,  10/1/2019
White Papers
Register for InformationWeek Newsletters
State of the Cloud
State of the Cloud
Cloud has drastically changed how IT organizations consume and deploy services in the digital age. This research report will delve into public, private and hybrid cloud adoption trends, with a special focus on infrastructure as a service and its role in the enterprise. Find out the challenges organizations are experiencing, and the technologies and strategies they are using to manage and mitigate those challenges today.
Current Issue
Getting Started With Emerging Technologies
Looking to help your enterprise IT team ease the stress of putting new/emerging technologies such as AI, machine learning and IoT to work for their organizations? There are a few ways to get off on the right foot. In this report we share some expert advice on how to approach some of these seemingly daunting tech challenges.
Flash Poll