Apache Spark Updates, Executive Data Strategies: Big Data Roundup - InformationWeek
Data Management
10:05 AM
Connect Directly

Apache Spark Updates, Executive Data Strategies: Big Data Roundup

In our big data roundup for the week of Nov. 27, we've got updates on Apache Spark and its ecosystem, a reality check on the practical use of artificial intelligence today and more.

CES 2016 Sneak Peek: 9 Cool Gadgets
CES 2016 Sneak Peek: 9 Cool Gadgets
(Click image for larger view and slideshow.)

You might expect a slow week for big data news as all the analysts and data scientists focused on optimizing meal preparation, predicting football scores, and surfacing the best prices on the most sought after holiday gifts. But you'd be wrong. This week we had news from IBM and Databricks about Apache Spark, a closer look at how a couple of different companies -- AccuWeather and McGraw-Hill Education -- have built analytics into the services they offer, a great primer on Artificial Intelligence and the future, and more.

Let's start with the IBM and Databricks announcements around Apache Spark.

You may remember that IBM Analytics contributed its SystemML engine for machine learning to the Apache Foundation earlier this year. IBM said the goal was to improve upon Spark's MLlib machine learning algorithms and libraries. This week IBM announced that its SystemML has been accepted into the Apache Incubator program.

(Image: PonyWang/iStockphoto)

(Image: PonyWang/iStockphoto)

Also this week, Databricks started previewing its software-as-a-service implementation of the next version of big data platform Apache Spark. Spark 1.6.0 is not due for general release until mid-December, but big data pros can take it for a test run on Databricks starting now. The new version is focused on improving performance. Check out the full story on Apache Spark's updates this week here.

What kind of company has been in the business of making predictions from data for a really long time already? How about companies that predict the weather. InformationWeek recently spoke to AccuWeather's Chief Commercial Officer about how the company is leveraging its proprietary analytics systems and big data to offer what amounts to predictive analytics as a service for customers. Retailers are big believers in this, but other industries are getting on board, too.

Our InformationWeek story has all the details.

We also spoke this week to the Chief Digital Officer of McGraw-Hill Education about how his company is looking at student click streams on digital curriculum and leveraging them to improve outcomes for individual students. The company is also using them to help instructors gain insights on how to best help students and to improve its own curriculum. 

Artificial Intelligence is a big topic in the industry and among science fiction writers, but how much do you really know about how it's being used today? InformationWeek recently connected with an expert on this technology for a reality check.

Meanwhile, it appears that if you are a data scientist (or studying to be one) you're in a great place in terms of salary. Data scientist is one of the job titles that is getting a big salary bump in 2016, according to our list here. Click through to find out just how much of an increase, and what other job titles can expect rich rewards next year.

[IBM Watson wants to help you leverage analytics to get the jump on hot gifts for the holidays. Read IBM Watson Trend App: Big Data Meets Holiday Shopping.]

In this shortened week, we did not have time to cover a couple of new partnerships announced by Dell Services around healthcare analytics. The company signed a multi-year agreement with Zebra Medical Vision to deliver a platform for medical imaging research. Dell Services also announced the integration of cloud-based analytics software BizEye to the Dell Cloud Clinical Archive portal.

We may have mentioned this one before, but since you are at the end of a long Thanksgiving Day weekend and perhaps still unsure about your holiday shopping strategy we thought we'd mention this one again. IBM Watson has created an iOS and web app to surface the hot gifts this holiday season. Don't know what to get for your kids, spouse, mother-in-law? You may find your answers, or at least some inspiration here

**New deadline of Dec. 18, 2015** Be a part of the prestigious InformationWeek Elite 100! Time is running out to submit your company's application by Dec. 18, 2015. Go to our 2016 registration page: InformationWeek's Elite 100 list for 2016.

Jessica Davis has spent a career covering the intersection of business and technology at titles including IDG's Infoworld, Ziff Davis Enterprise's eWeek and Channel Insider, and Penton Technology's MSPmentor. She's passionate about the practical use of business intelligence, ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
12/8/2015 | 5:48:48 AM
Re: SystemML is an algorithm processing engine
I learned a lot by reading your post but I'm just asking myself a question : A friend of I used to speak about Apache Cassandra, what is the difference between Cassandra and Spark for Big Data ? 
User Rank: Ninja
12/5/2015 | 4:42:21 PM
Re: SystemML is an algorithm processing engine
@Charlie - thanks for sharing, as technology changing... it not easy to keep up
Charlie Babcock
Charlie Babcock,
User Rank: Author
11/30/2015 | 4:09:41 PM
SystemML is an algorithm processing engine
During IBM's DataPalooza in San Francisco, IBM VP of Analytics Product Rob Thomas explained to IW that SystemML is an engine for running machine learning algorithms, such as those found in Spark's MLib library. The engine takes advantage of a server cluster to distribute work and process results efficiently. Machine learning frequently requires a scale-out architecture.
2018 State of the Cloud
2018 State of the Cloud
Cloud adoption is growing, but how are organizations taking advantage of it? Interop ITX and InformationWeek surveyed technology decision-makers to find out, read this report to discover what they had to say!
A Data-Centric Approach to the US Census
James M. Connolly, Executive Managing Editor, InformationWeekEditor in Chief,  10/12/2018
10 Top Strategic Predictions for 2019
Jessica Davis, Senior Editor, Enterprise Apps,  10/17/2018
AI & Machine Learning: An Enterprise Guide
James M. Connolly, Executive Managing Editor, InformationWeekEditor in Chief,  9/27/2018
Register for InformationWeek Newsletters
Current Issue
The Next Generation of IT Support
The workforce is changing as businesses become global and technology erodes geographical and physical barriers.IT organizations are critical to enabling this transition and can utilize next-generation tools and strategies to provide world-class support regardless of location, platform or device
White Papers
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Sponsored Video
Flash Poll