Apache Spark Ignites Big Data Adoption - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Data Management // Big Data Analytics
News
2/10/2016
10:06 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Apache Spark Ignites Big Data Adoption

Apache Spark is among the Hadoop ecosystem technologies acting as catalysts for broader adoption of big data infrastructure. Now, Looker -- a vendor of business intelligence software -- has announced support for Spark and other Hadoop technologies. The goal? To speed up access to the data that fuels business decision making.

Hadoop At 10: Milestones And Momentum
Hadoop At 10: Milestones And Momentum
(Click image for larger view and slideshow.)

Hadoop's arrival on the scene 10 years ago may have started the big data revolution, but only recently did adoption of this technology begin spreading to a wider audience. Apache Spark is one of the catalysts for the growing adoption rates.

Spark can be used as a replacement for MapReduce, a component of Hadoop implementations, to speed up the processing and analytics of big data by 100x in memory, according to the Apache Software Foundation.

In today's business environment, in which real-time analytics is the goal and organizations don't want to wait for data warehouses and analysts to provide batch intelligence back to business users, Spark has gained momentum.

[ Looking for more on Spark's momentum? Read IBM, Databricks Advance Apache Spark. ]

And here's one case in point: Looker, a business intelligence platform used by Avant, Acorns, and Etsy, this week announced support for Presto and Spark SQL. The company also updated its support for Impala and Hive, other Hadoop ecosystem technologies that speed up analysis on Hadoop.

Looker's announcement of support for these additional Hadoop ecosystem technologies lets organizations "leave data in Hadoop and process it at speed and at scale," said James Haight, principal analyst at Blue Hill Research, in an interview with InformationWeek.

(Image: matdesign24/iStockphoto)

(Image: matdesign24/iStockphoto)

"What we are talking about is getting some real business value out of this big data infrastructure that's been talked about for so long," Looker CEO Frank Bien told InformationWeek in an interview. "Hadoop is finally moving beyond the science experiments. Organizations can query large amounts of data."

That's what Facebook does with Presto, Bien noted. And that's what many other companies are doing with Spark.

Looker's announcement this week may be the first in a series as vendors get ready for the Spark Summit East in New York City, February 16-18. Speakers include technologists and executives from Databricks, IBM, Capital One, Hortonworks, SAP, and eBay.

Haight, of Blue Hill Research, said many of his clients have been storing much more data in recent years, largely generated by social media and the Internet of Things (IoT). "These have outpaced our ability to store data, let alone analyze it," he said. And now Hadoop has enabled businesses to store even more of it. 

"We are pouring data in, but we have no ability to process it. How do we process all this data? I'm dealing with a lot of companies at that juncture," Haight said.

Some companies have coped with the data overload by pulling out some data and "spoon feeding" it into other analytics systems, according to Haight. That's not an ideal approach in an age where everyone wants intelligence in real time.

Bien said "It is riding a giant evolutionary shift. That's what this is all about." It's not only about making data big. It's about making data fast. And it's also about making big data accessible to the business users, not just to a small group of analysts or IT workers.

"We see the business intelligence and analytics market becoming one with the big data market," Bien said, pointing to Gartner's recent redefinition of the business intelligence and analytics market. "This is the future of how people are going to interact with and store big data, and we are right at the core of it."

Are you an IT Hero? Do you know someone who is? Submit your entry now for InformationWeek's IT Hero Award. Full details and a submission form can be found here.

Jessica Davis has spent a career covering the intersection of business and technology at titles including IDG's Infoworld, Ziff Davis Enterprise's eWeek and Channel Insider, and Penton Technology's MSPmentor. She's passionate about the practical use of business intelligence, ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
pfretty
100%
0%
pfretty,
User Rank: Ninja
2/12/2016 | 9:07:29 PM
Growing trend
A recent IDG study showed the movement toward Hadoop is continuing to grow with 30 percent now seeking Hadoop solutions for unstructured data storage. However, IT leaders are also clear that its necessary to augment Hadoop to enjoy its true benefits. The full results are available at the Big Data Forum. Peter Fretty, IDG blogger for SAS
Slideshows
What Digital Transformation Is (And Isn't)
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/4/2019
Commentary
Watch Out for New Barriers to Faster Software Development
Lisa Morgan, Freelance Writer,  12/3/2019
Commentary
If DevOps Is So Awesome, Why Is Your Initiative Failing?
Guest Commentary, Guest Commentary,  12/2/2019
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Getting Started With Emerging Technologies
Looking to help your enterprise IT team ease the stress of putting new/emerging technologies such as AI, machine learning and IoT to work for their organizations? There are a few ways to get off on the right foot. In this report we share some expert advice on how to approach some of these seemingly daunting tech challenges.
Slideshows
Flash Poll