Apache Spark Updates, Executive Data Strategies: Big Data Roundup - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management
10:05 AM
Connect Directly

Apache Spark Updates, Executive Data Strategies: Big Data Roundup

In our big data roundup for the week of Nov. 27, we've got updates on Apache Spark and its ecosystem, a reality check on the practical use of artificial intelligence today and more.

CES 2016 Sneak Peek: 9 Cool Gadgets
CES 2016 Sneak Peek: 9 Cool Gadgets
(Click image for larger view and slideshow.)

You might expect a slow week for big data news as all the analysts and data scientists focused on optimizing meal preparation, predicting football scores, and surfacing the best prices on the most sought after holiday gifts. But you'd be wrong. This week we had news from IBM and Databricks about Apache Spark, a closer look at how a couple of different companies -- AccuWeather and McGraw-Hill Education -- have built analytics into the services they offer, a great primer on Artificial Intelligence and the future, and more.

Let's start with the IBM and Databricks announcements around Apache Spark.

You may remember that IBM Analytics contributed its SystemML engine for machine learning to the Apache Foundation earlier this year. IBM said the goal was to improve upon Spark's MLlib machine learning algorithms and libraries. This week IBM announced that its SystemML has been accepted into the Apache Incubator program.

(Image: PonyWang/iStockphoto)

(Image: PonyWang/iStockphoto)

Also this week, Databricks started previewing its software-as-a-service implementation of the next version of big data platform Apache Spark. Spark 1.6.0 is not due for general release until mid-December, but big data pros can take it for a test run on Databricks starting now. The new version is focused on improving performance. Check out the full story on Apache Spark's updates this week here.

What kind of company has been in the business of making predictions from data for a really long time already? How about companies that predict the weather. InformationWeek recently spoke to AccuWeather's Chief Commercial Officer about how the company is leveraging its proprietary analytics systems and big data to offer what amounts to predictive analytics as a service for customers. Retailers are big believers in this, but other industries are getting on board, too.

Our InformationWeek story has all the details.

We also spoke this week to the Chief Digital Officer of McGraw-Hill Education about how his company is looking at student click streams on digital curriculum and leveraging them to improve outcomes for individual students. The company is also using them to help instructors gain insights on how to best help students and to improve its own curriculum. 

Artificial Intelligence is a big topic in the industry and among science fiction writers, but how much do you really know about how it's being used today? InformationWeek recently connected with an expert on this technology for a reality check.

Meanwhile, it appears that if you are a data scientist (or studying to be one) you're in a great place in terms of salary. Data scientist is one of the job titles that is getting a big salary bump in 2016, according to our list here. Click through to find out just how much of an increase, and what other job titles can expect rich rewards next year.

[IBM Watson wants to help you leverage analytics to get the jump on hot gifts for the holidays. Read IBM Watson Trend App: Big Data Meets Holiday Shopping.]

In this shortened week, we did not have time to cover a couple of new partnerships announced by Dell Services around healthcare analytics. The company signed a multi-year agreement with Zebra Medical Vision to deliver a platform for medical imaging research. Dell Services also announced the integration of cloud-based analytics software BizEye to the Dell Cloud Clinical Archive portal.

We may have mentioned this one before, but since you are at the end of a long Thanksgiving Day weekend and perhaps still unsure about your holiday shopping strategy we thought we'd mention this one again. IBM Watson has created an iOS and web app to surface the hot gifts this holiday season. Don't know what to get for your kids, spouse, mother-in-law? You may find your answers, or at least some inspiration here

**New deadline of Dec. 18, 2015** Be a part of the prestigious InformationWeek Elite 100! Time is running out to submit your company's application by Dec. 18, 2015. Go to our 2016 registration page: InformationWeek's Elite 100 list for 2016.

Jessica Davis is a Senior Editor at InformationWeek. She covers enterprise IT leadership, careers, artificial intelligence, data and analytics, and enterprise software. She has spent a career covering the intersection of business and technology. Follow her on twitter: ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Charlie Babcock
Charlie Babcock,
User Rank: Author
11/30/2015 | 4:09:41 PM
SystemML is an algorithm processing engine
During IBM's DataPalooza in San Francisco, IBM VP of Analytics Product Rob Thomas explained to IW that SystemML is an engine for running machine learning algorithms, such as those found in Spark's MLib library. The engine takes advantage of a server cluster to distribute work and process results efficiently. Machine learning frequently requires a scale-out architecture.
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Blockchain Gets Real Across Industries
Lisa Morgan, Freelance Writer,  7/22/2021
Seeking a Competitive Edge vs. Chasing Savings in the Cloud
Joao-Pierre S. Ruth, Senior Writer,  7/19/2021
How CIO Roles Will Change: The Future of Work
Jessica Davis, Senior Editor, Enterprise Apps,  7/1/2021
White Papers
Register for InformationWeek Newsletters
2021 State of ITOps and SecOps Report
2021 State of ITOps and SecOps Report
This new report from InformationWeek explores what we've learned over the past year, critical trends around ITOps and SecOps, and where leaders are focusing their time and efforts to support a growing digital economy. Download it today!
Current Issue
Monitoring Critical Cloud Workloads Report
In this report, our experts will discuss how to advance your ability to monitor critical workloads as they move about the various cloud platforms in your company.
Flash Poll