IoT
IoT
Data Management // Software Platforms
News
11/25/2015
09:35 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

IBM, Databricks Advance Apache Spark

Apache Spark hit two more incremental milestones as the IBM SystemML machine learning engine for Spark was accepted to the Apache Incubator program and Databricks unveiled a preview of its SaaS edition of Spark 1.6.

Data Science Skills To Boost Your Salary
Data Science Skills To Boost Your Salary
(Click image for larger view and slideshow.)

Apache Spark momentum continued this week as IBM's SystemML machine learning engine for Spark won acceptance as a project by the open source Apache Incubator. Separately, Databricks, the company whose founders were the original architects of Spark at the University of California, Berkeley's AMPLab, started showing off a preview of Spark 1.6 in the company's software-as-a-service implementation of the platform.

Spark 1.6 is due for general release in mid-December.

The announcements from these two influential companies in the Spark community built on what's already been fast-growing support for the big data platform. According to a Spark users survey, and information released last month by Databricks about Spark momentum, Spark is the most active open source project in all of big data, with over 600 contributors in the last 12 months, up from 314 in the previous 12 to 24 months. What's driving the widespread support for this year-old platform?

Rob Thomas, IBM Analytics vice president of product, compared Spark's expected utility and ultimate impact to that of the Linux operating system.

"The fact that Spark had a single programming model and the ability to analyze all types of data from all sources of data positioned it to have the impact in the industry that something like Linux did at the turn of the century," Thomas told InformationWeek in an interview. "Linux is an operating system for systems and computers. Spark will be the operating system around analytics and how data will be accessed." That's how big and important IBM thinks Spark will become.

(Image: NorthernStock/iStockphoto)

(Image: NorthernStock/iStockphoto)

That's also why IBM this week was celebrating the acceptance of its SystemML into the Apache Incubator.

IBM first announced plans to donate SystemML to the Spark ecosystem in June as part of a big commitment the company made to the open source project. IBM developed SystemML to complement Spark's MLlib, a set of libraries or algorithms in Spark that can be used to analyze a set of data. But Thomas has said that Spark's machine learning is the weakest part of the platform. IBM built and then donated SystemML to improve that component of Spark.

"Our contribution of SystemML was about making Spark better," he said. "It made sense for us to go down that route. We've are a huge supporter of Spark to date."

IBM last month added Spark to its BlueMix cloud platform.

[Need more background on Apache Spark? Read Spark Promoter Databricks Should Let Software Shine.]

Separately this week, Databricks began a preview of the newest version of Spark, 1.6.0, that customers can look at on the company's SaaS Spark platform now. The general release of Spark 1.6.0 is expected in mid-December.

The new version improves two things that are hugely important to Spark users -- performance and optimization -- Databricks cofounder Patrick Wendell told InformationWeek in an interview.

"Performance is a major theme of this release," Wendell said. Previous versions offered users a series of recommendations and settings to use to get the best performance out of memory. Such fine tuning for high performance will no longer be necessary.

"For most users, if they upgrade to this release they will experience a huge performance increase," Wendell said. "Users don't need to tune this at all."

The new version also includes a new API called the Dataset API. This is an extension of Spark's DataFrame API that enables developers to write programs much more concisely with far less code, Wendell said.

Another new big feature is optimized state storage in Spark Streaming, Wendell said. That can translate to a "10x performance gain for many workloads," Wendell wrote in a blog post about the new release this week.

Databricks has further details of the new release in the blog post and has scheduled a December 1 webinar to go over the improvements and changes.

**New deadline of Dec. 18, 2015** Be a part of the prestigious InformationWeek Elite 100! Time is running out to submit your company's application by Dec. 18, 2015. Go to our 2016 registration page: InformationWeek's Elite 100 list for 2016.

Jessica Davis has spent a career covering the intersection of business and technology at titles including IDG's Infoworld, Ziff Davis Enterprise's eWeek and Channel Insider, and Penton Technology's MSPmentor. She's passionate about the practical use of business intelligence, ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
batye
50%
50%
batye,
User Rank: Ninja
12/5/2015 | 9:55:24 PM
interesting info
interesting to know, thanks for sharing... good info 
In A Fever For Big Data
In A Fever For Big Data
Healthcare orgs are relentlessly accumulating data, and a growing array of tools are becoming available to manage it.
Register for InformationWeek Newsletters
White Papers
Current Issue
Top IT Trends to Watch in Financial Services
IT pros at banks, investment houses, insurance companies, and other financial services organizations are focused on a range of issues, from peer-to-peer lending to cybersecurity to performance, agility, and compliance. It all matters.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of August 21, 2016. We'll be talking with the InformationWeek.com editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.