MapReduce: And You Were There - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Software // Information Management
Commentary
8/29/2008
09:01 AM
Neil Raden
Neil Raden
Commentary
50%
50%

MapReduce: And You Were There

There's been a lot of buzz lately about Google's MapReduce framework for speeding up the processing of large datasets. It makes you wonder, did Google just dream this up in last couple years while all of the database vendors were sleeping? Or, paraphrasing Isaac Newton, were they standing on the shoulders of giants? The answer is, both.

There's been a lot of buzz lately about Google's MapReduce framework for speeding up the processing of large datasets. It makes you wonder, did Google just dream this up in last couple years while all of the database vendors were sleeping? Or, paraphrasing Isaac Newton, were they standing on the shoulders of giants?

The answer is, both.MapReduce is a programming framework, not a language per se. It is built on an old (40+ years) programming paradigm called functional programming (just for the record, the other type of paradigm is called imperative programming and includes common languages like C# and Java). Maybe I shouldn't have said old, because my first programming language was an early functional language, APL. I was a casualty actuary and APL was perfect for doing the kinds of mathematical manipulations we needed to do, such as matrix inversion in one keystroke, recursion and manipulating n-dimensional structures with composite functions. We used to drive IT nuts. Functional languages operate on, obviously, mathematical functions and some well-known functional languages today include the successor to APL, K and the statistical language R.

The separation of functional and imperative languages is pretty leaky these days as lots of functional programming ideas have seeped into other languages. In particular, the concepts of map and reduce are widely implemented. So why, then, does it matter what you use?

The symbolic language and its syntax, rules and scope have a lot to do with what programmers can achieve and how easily they can do it, but computers don't execute symbolic code, it has to be turned into instructions that a computer (or a whole bunch of computers) can understand. If every language just gets reduced to this level, you might wonder what the difference is. The real advantage is in the compiler. In a functional language, the map function, for example, when used in composition (putting functions together) can eliminate a second, expensive map by understanding them together at compilation. The compiler designer, working from a purely functional position, can develop compilations that really leverage the symbolic language.

And this is where Google has had breakthroughs. They had to approach this problem as a fundamental aspect of doing business and developed some creative ways to really power through sets of data, but they didn't do it alone. Computer scientists have been advancing these ideas for decades.There's been a lot of buzz lately about Google's MapReduce framework for speeding up the processing of large datasets. It makes you wonder, did Google just dream this up in last couple years while all of the database vendors were sleeping? Or, paraphrasing Isaac Newton, were they standing on the shoulders of giants? The answer is, both.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Commentary
Why It's Nice to Know What Can Go Wrong with AI
James M. Connolly, Editorial Director, InformationWeek and Network Computing,  11/11/2019
Slideshows
Top-Paying U.S. Cities for Data Scientists and Data Analysts
Cynthia Harvey, Freelance Journalist, InformationWeek,  11/5/2019
Slideshows
10 Strategic Technology Trends for 2020
Jessica Davis, Senior Editor, Enterprise Apps,  11/1/2019
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Getting Started With Emerging Technologies
Looking to help your enterprise IT team ease the stress of putting new/emerging technologies such as AI, machine learning and IoT to work for their organizations? There are a few ways to get off on the right foot. In this report we share some expert advice on how to approach some of these seemingly daunting tech challenges.
Slideshows
Flash Poll