Apache Spark Updates, Executive Data Strategies: Big Data Roundup

In our big data roundup for the week of Nov. 27, we've got updates on Apache Spark and its ecosystem, a reality check on the practical use of artificial intelligence today and more.

Jessica Davis, Senior Editor

November 29, 2015

4 Min Read
<p align="left">(Image: PonyWang/iStockphoto)</p>

CES 2016 Sneak Peek: 9 Cool Gadgets

CES 2016 Sneak Peek: 9 Cool Gadgets


CES 2016 Sneak Peek: 9 Cool Gadgets (Click image for larger view and slideshow.)

You might expect a slow week for big data news as all the analysts and data scientists focused on optimizing meal preparation, predicting football scores, and surfacing the best prices on the most sought after holiday gifts. But you'd be wrong. This week we had news from IBM and Databricks about Apache Spark, a closer look at how a couple of different companies -- AccuWeather and McGraw-Hill Education -- have built analytics into the services they offer, a great primer on Artificial Intelligence and the future, and more.

Let's start with the IBM and Databricks announcements around Apache Spark.

You may remember that IBM Analytics contributed its SystemML engine for machine learning to the Apache Foundation earlier this year. IBM said the goal was to improve upon Spark's MLlib machine learning algorithms and libraries. This week IBM announced that its SystemML has been accepted into the Apache Incubator program.

Also this week, Databricks started previewing its software-as-a-service implementation of the next version of big data platform Apache Spark. Spark 1.6.0 is not due for general release until mid-December, but big data pros can take it for a test run on Databricks starting now. The new version is focused on improving performance. Check out the full story on Apache Spark's updates this week here.

What kind of company has been in the business of making predictions from data for a really long time already? How about companies that predict the weather. InformationWeek recently spoke to AccuWeather's Chief Commercial Officer about how the company is leveraging its proprietary analytics systems and big data to offer what amounts to predictive analytics as a service for customers. Retailers are big believers in this, but other industries are getting on board, too.

Our InformationWeek story has all the details.

We also spoke this week to the Chief Digital Officer of McGraw-Hill Education about how his company is looking at student click streams on digital curriculum and leveraging them to improve outcomes for individual students. The company is also using them to help instructors gain insights on how to best help students and to improve its own curriculum. 

Artificial Intelligence is a big topic in the industry and among science fiction writers, but how much do you really know about how it's being used today? InformationWeek recently connected with an expert on this technology for a reality check.

Meanwhile, it appears that if you are a data scientist (or studying to be one) you're in a great place in terms of salary. Data scientist is one of the job titles that is getting a big salary bump in 2016, according to our list here. Click through to find out just how much of an increase, and what other job titles can expect rich rewards next year.

[IBM Watson wants to help you leverage analytics to get the jump on hot gifts for the holidays. Read IBM Watson Trend App: Big Data Meets Holiday Shopping.]

In this shortened week, we did not have time to cover a couple of new partnerships announced by Dell Services around healthcare analytics. The company signed a multi-year agreement with Zebra Medical Vision to deliver a platform for medical imaging research. Dell Services also announced the integration of cloud-based analytics software BizEye to the Dell Cloud Clinical Archive portal.

We may have mentioned this one before, but since you are at the end of a long Thanksgiving Day weekend and perhaps still unsure about your holiday shopping strategy we thought we'd mention this one again. IBM Watson has created an iOS and web app to surface the hot gifts this holiday season. Don't know what to get for your kids, spouse, mother-in-law? You may find your answers, or at least some inspiration here

**New deadline of Dec. 18, 2015** Be a part of the prestigious InformationWeek Elite 100! Time is running out to submit your company's application by Dec. 18, 2015. Go to our 2016 registration page: InformationWeek's Elite 100 list for 2016.

About the Author

Jessica Davis

Senior Editor

Jessica Davis is a Senior Editor at InformationWeek. She covers enterprise IT leadership, careers, artificial intelligence, data and analytics, and enterprise software. She has spent a career covering the intersection of business and technology. Follow her on twitter: @jessicadavis.

Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.

You May Also Like


More Insights