Micosoft's SQL Strategy For Massive Data Sets - InformationWeek
Software // Information Management
08:10 PM
Roger Smith
Roger Smith
Building Security for the IoT
Nov 09, 2017
In this webcast, experts discuss the most effective approaches to securing Internet-enabled system ...Read More>>

Micosoft's SQL Strategy For Massive Data Sets

Cloud computing service providers like Microsoft, Google, and Yahoo are all hard at work on a new generation of parallel data processing tools that will make it easier for each company to store and analyze enormous data sets such as search logs and click streams.

Cloud computing service providers like Microsoft, Google, and Yahoo are all hard at work on a new generation of parallel data processing tools that will make it easier for each company to store and analyze enormous data sets such as search logs and click streams.One of the more interesting papers presented at this week's VLDB (Very Large Data Base) conference in Auckland, New Zealand, "Scope: Easy and Efficient Parallel Processing of Massive Data Sets," (PDF), describes one particular parallel data processing tool developed by Microsoft Research that's being used daily inside Microsoft over petabytes of data on large clusters of thousands of commodity servers, including those Microsoft will use to equip the new $500 million data center the Redmond,Wash.-based company is building in West Des Moines, Iowa.

Microsoft's Scope is similar to Yahoo's Pig, which is a higher-level language on top of Yahoo's Hadoop distributed software framework, or Google's Sawzall, which is a higher-level language on top of the MapReduce framework and the Google File System. According to the VLDB08 paper's authors, Ronnie Chaiken, Bob Jenkins, Per-Ake Larson, Bill Ramsey, Darren Shakib, Simon Weaver, and Jingren Zhou, where Pig and Sawzall promote a more functional or mathematical programming style, Scope looks much more like SQL.

"SCOPE has a strong resemblance to SQL -- an intentional design choice. . . Users familiar with SQL require little or no training to use Scope. Like SQL, data is modeled as a set of rows composed of typed columns. Every rowset has a well-defined schema ... It allows users to focus on the data transformations required to solve the problem at hand and hides the complexity of the underlying platform and implementation details."

The language is high-level and declarative so that the Scope compiler and optimizer can optimize Scope scripts and improve them over time. All the hardware and implementation details are transparent to users.

According to the paper, Scope is highly extensible. Users can easily create customized operators, including:

"• extractors (for parsing and constructing rows from a file), • processors (for row-wise processing), • reducers (for group-wise processing), and • combiners (for combining rows from two inputs) ...

[all of which] allow users to solve problems that cannot easily be expressed in traditional SQL."

All companies that operate Internet-scale services have the need to store and process massive data sets, such as search logs, Web content collected by crawlers, and click-streams collected from a variety of Web services. Google, Yahoo, and Microsoft have developed their own systems that support parallel computations over large (multiple petabyte) data sets on clusters of computers. Google popularized the map-reduce programming model, largely taken from the map and reduce functions commonly used in a functional or mathematical style of programming. Yahoo also has a software stack designed for distributed processing of massive data sets. Users write applications in a language called Pig Latin, which is a dataflow language that uses a nested data model. A Pig Latin program is compiled by the Pig system into a sequence of MapReduce operators that are executed using Hadoop, an open-source implementation of MapReduce.

When Pigs Have Wings

According to the Microsoft paper, a MapReduce application written in C++ takes many more lines of code than the corresponding application expressed in Scope; giving an example that requires 70 lines of C++ code but only six lines of Scope code.

Analysis of massive data sets is becoming increasingly valuable for businesses like Microsoft, in order to support new features and do things like improve service quality and detect changes in patterns over time that can detect fraudulent activity. The new Scope (Structured Computations Optimized for Parallel Execution) language is targeted for large-scale data analysis under development at Microsoft. Scope has the advantage of intentionally building on end-user knowledge of relational data and SQL, with some simplifications that ought to make it easier for the company to take advantage of the new parallel processing execution environment the software giant continues to relentlessly build.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of IT Report
In today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll