YARN: Answer To Hadoop's Shortcomings?

Apache Hadoop is certainly not the only open source distributed file system solution to emerge in recent years -- competitors such as HPCC are out there as well -- but it's certainly the best-known and most popular platform in the burgeoning big data space.

Despite its adeptness at bringing data processing and analysis to raw storage, however, Hadoop has its shortcomings, such as batch processing delays and its reliance on MapReduce for data processing.

YARN, which stands for Yet Another Resource Negotiator, is a new framework that Cloudera calls "more generic than the earlier MapReduce implementation," in that it runs programs that don't follow the MapReduce model.

"In a nutshell, YARN is our attempt to take Hadoop beyond just MapReduce for data processing. What YARN allows you to do is do much more than MapReduce in the same Hadoop cluster," Arun Murthy, chair of Apache Hadoop Project Management Committee and one of Hortonworks' founders, recently told InformationWeek.

[ Read about Pervasive Software's powerful visual tools. See Big Data's 'Wild West' Period Stars Hadoop. ]

Pervasive Software chief technologist Jim Falgout uses this analogy: Think of YARN as a distributed operating system -- "which it is," he told InformationWeek in a phone interview. "It has a distributed scheduler, a distributed file system. And it allows you to run jobs that are distributed."

But pre-YARN Hadoop is "basically an operating system that allows you to run one application only," Falgout chuckled. "That's MapReduce." He added, "YARN is a really important part of Hadoop going forward. It shows a lot of maturity in the Hadoop community to say that MapReduce is great, but it doesn't solve all the problems. It's not for every solution that you want to build."

Falgout has addressed this topic on other forums as well. "MapReduce is great at solving problems such as indexing the world's web sites. However, it's not the most flexible or the most efficient platform for every computation problem out there," he wrote in a recent blog post on the Data Science Central site.

YARN opens up Hadoop, allowing developers to build different types of applications that take advantage of the platform's attributes. "If you want to work with Hadoop, pre-YARN, you have to write all of your applications on this nice, distributed operating system in MapReduce," Falgout said. "If Linux was built that way, it wouldn't have gotten very far."

Given Falgout's thoughts on the YARN, it's little surprise that Pervasive Software, an Austin, Texas-based provider of data management and analytics products, is deeply involved in Hadoop and big data. The company's software offerings include DataRush, which is designed to boost parallel performance of data preparation and analytics tasks, and RushAnalyzer, a visual workflow tool for data access, preparation, analysis and reporting.

Pervasive plans to port its DataRush framework to YARN, Falgout said, but has yet to announce a specific timetable. "We don't see YARN being used in a lot of customer production systems yet," he added. "We know it's coming, so we're going to start working on YARN probably within the next month."

Cloudera, meanwhile, has implemented an early version of YARN in its CDH4 distribution of Hadoop, but the software is considered an alpha release. "It's out there. You can look at it," said Falgout. "You can play with it, but it's not ready for prime time."

YARN's implementation wouldn't directly impact end users, of course, but it could offer indirectly benefits that make the notoriously difficult Hadoop platform easier to use. "That was kind of a big theme at Strata Hadoop World recently: Making Hadoop more adoptable, making it easier to use," Falgout said. "Getting it out beyond the early adopters who were willing to write MapReduce code, and go through all that pain because they were getting a big benefit out of it."

Predictive analysis is getting faster, more accurate and more accessible. Combined with big data, it's driving a new age of experiments. Also in the new, all-digital Advanced Analytics issue of InformationWeek: Are project management offices a waste of money? (Free registration required.)