Microsoft Releases Hadoop On Windows

The big data world knew it was coming, but now we know the official name of Microsoft's Hadoop software distribution: Microsoft HDInsight Server for Windows. The vendor announced the first preview release of the on-premises software on Wednesday at the Strata Conference in New York.

Microsoft also announced the third release of its Azure cloud-based Hadoop service, now known as Windows Azure HDInsight Service. Both were developed in partnership with Hadoop software developer and distributor Hortonworks.

HDInsight Server for Windows includes core Hadoop (meaning the Hadoop Distributed File System and MapReduce), plus Pig for MapReduce programming, Hive data query infrastructure, Hortonworks' recently introduced HCatalog table management service for access to Hadoop data, Scoop for data movement, and the Ambari monitoring and management console. All of the above have been reengineered to run on Windows and all are open-source components that are compatible with Apache Hadoop and are being contributed back to the community.

[ Want more on Hortonworks' software? Read Hortonworks' Hadoop Dilemma: Get Rich Giving Ideas Away. ]

The HDInsight Server is designed to work with (but does not include) Windows Server and Microsoft SQL Server. In the case of Windows, HDInsight is integrated with Microsoft System Center for administrative control and Active Directory for access control and security.

"We've spent tremendous engineering time integrating it with the Windows management platform, so we end up with enterprise-grade, enterprise-scale management, deployment, and integrated security with the Windows infrastructure environment," said Doug Leland, Microsoft's general manager of SQL Server Marketing, in an interview with InformationWeek.

The integration with the larger systems environment is a contrast with the Oracle Big Data Appliance, which hands all things Hadoop over to Cloudera's distribution of Apache Hadoop and proprietary Cloudera Manager administrative software (the appliance runs on Oracle Enterprise Linux, but deployment, admin, and user access is managed by Cloudera's software).

HDInsight Server for Windows also works with Microsoft's virtualization platform, including Hyper-V and the System Center virtualization fabric. "We're dramatically reducing what it takes to provision and manage either your head node or your data node," said Leyland. "We're creating templates that let you define what your head node and data nodes look like, and then we're using virtualization to dynamically spin up and provision nodes across however many instances that you want."

Microsoft is the first to deliver Hadoop on virtualized infrastructure, according to Leyland, but it won't be alone. Project Serengeti is an open-source effort designed to support rapid deployment, configuration, and management of a Hadoop cluster in a virtual environment. VMWare CTO Richard McDougall was slated to present on Serengeti at the Strata conference on Thursday.

The HDInsight (on-premises and cloud) connection to Microsoft SQL Server is all about business intelligence, and that starts with user-facing tools and components including Microsoft Excel, PowerPivot for Excel, and Power View. Few people don't have access to Excel, and it can handle data extracts from any Hadoop environment.

The PowerPivot add-in for high-scale in-memory data exploration and Power View component for data visualization are available as free downloads from microsoft, but they have to be licensed to run against Microsoft SQL Server through a client access license. In Office 2013, which is currently in preview release, PowerPivot and Power View are natively integrated with Excel.

How can you explore Hadoop data using Microsoft's Excel-based BI tools? There are currently two options: using Hortonworks' previously available Hive ODBC driver to access Hadoop data directly or using Scoop connectors to Hadoop (also previously available) to move data out of Hadoop and into the Microsoft SQL Server database or into SQL Server Analysis Services.

Microsoft also plans to make use of HCatalog metadata services, Leland said, but he was uncertain about what level of integration had already been achieved. Hortonworks partner Teradata is already using HCatalog, which is designed to enable users to define the structure and location of data within Hadoop to support fast and consistent access and analysis without actually moving data.

Microsoft's cloud-based HDInsight for Azure service, which is now in its third preview release, lets users spin up and deploy a Hadoop cluster within minutes, according to Leland. This service will compete with existing MapReduce services including Amazon Web Services' Elastic Map Reduce, which was introduced in April, 2009.

Microsoft did not disclose when the HDInsight Server for Windows Server and HDInsight For Azure service would be generally released, but it's Microsoft's usual practice to go through three community technology previews (each spanning several months) before making software generally available.

Since this is the first preview release of the on-premises software, which can be downloaded at no cost, it's a safe bet HDInsight Server for Windows won't be generally available before the second quarter of next year. The Azure cloud service is in the last preview stage, so it could be available by the end of this year or early next year.