Self-Service Access, Cloud Drive Big Data Value - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management
07:00 AM
Connect Directly

Self-Service Access, Cloud Drive Big Data Value

Data governance has emerged as a top concern, and business intelligence remains the top workload for big data technology Hadoop, according to the third edition of the Big Data Maturity Survey.

Business intelligence is still the biggest -- and growing -- workload planned for big data. And organizations are also adding more self-service access to big data and putting more big data in the cloud, all positive trends for getting value from investments in this technology.

Credit: Pixabay
Credit: Pixabay

That's according to the most recent edition of the Big Data Maturity Survey, conducted by BI-on-Hadoop vendor AtScale, together with Tableau, Trifacta, Cognizant, and the three major Hadoop distributors Cloudera, Hortonworks, and MapR. The survey is based on 2,550 responses from big data pros at 1,400 companies in 77 countries.

But there's a bit of a disconnect when it comes to the tangible value achieved in big data projects, with the likelihood of value depending on who is driving their adoption. While IT organizations drove 57% of big data projects, business users drove 33%, and an executive mandate drove 11%, projects that had the executive mandate as a driver were most likely to achieve tangible value -- 56%.

When IT was the driver 47% achieved tangible value, according to the survey results. For business user-driven big data projects, 52% achieved tangible value.

BI Continues To Grow

The survey shows a full 75% of respondents saying they plan to use big data for BI, up from the 69% who said the same thing last year. The results mark the continued momentum of BI workloads on big data platforms such as Hadoop. Last year's survey found that BI had overtaken ETL and data science as the top workload planned for Hadoop and related big data platforms.

This year's survey shows growth in cloud deployments for big data, too, with a full 53% saying they are using cloud for their big data deployment now, and 72% saying they plan on doing big data in the cloud in the future.

What's more, while 27% of respondents say they have more than half of their big data in the cloud today, 42% say they are planning to have more than half of their big data operations in the cloud in the future.

"There's been a clear surge in use of big data in the cloud over the last year, and what's perhaps as interesting is the fact that respondents are far more likely to achieve tangible value when their data is in the cloud," said AtScale co-founder and CTO Matt Baird in a prepared statement.

Baird is citing another finding of the survey there. The survey found that respondents who have deployed all of their big data in the cloud are 34% more likely to achieve tangible value than those who have none of their big data in the cloud.

More companies are also providing users with self-service access to big data, according to this year's survey. A full 47% of companies this year offered users self-service access, marking a 15% increase from last year. And those who offered self-service access this year were more likely to have achieved tangible value from their big data investments, 57%, compared to 43% at the companies that didn't offer self-service access.

Hadoop And Spark

More organizations are using Hadoop now than before -- 73% compared to last year's 65%.

But Apache Spark, while it may have a very vocal group of enthusiasts among technologists, is largely still in the experimental phase today in organizations surveyed.

While the majority of organizations -- 60% -- are still only using Apache Spark for educational purposes, organizations that have deployed Spark in production are 85% more likely to achieve value, according to the report.

Big Data's Role In Enterprise

The survey also asked about big data's role in enterprise data warehouses, and only 18% of respondents viewed big data as a replacement for the enterprise data warehouse. The rest, 82%, are looking to big data to augment and optimize the existing data warehouse rather than replacing it. That future data warehouse will combine traditional data warehousing solutions such as Teradata, with newer data platforms such as Hadoop, and with big data services such as Google BigQuery, Amazon Redshift, and Microsoft Azure, according to AtScale's report.

But organizations still expressed some concerns about big data going forward. The top three concerns cited by respondents were accessibility, security and governance, with worries related to governance growing the most at 21%, according to the report.

Jessica Davis is a Senior Editor at InformationWeek. She covers enterprise IT leadership, careers, artificial intelligence, data and analytics, and enterprise software. She has spent a career covering the intersection of business and technology. Follow her on twitter: ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

New Storage Trends Promise to Help Enterprises Handle a Data Avalanche
John Edwards, Technology Journalist & Author,  4/1/2021
11 Things IT Professionals Wish They Knew Earlier in Their Careers
Lisa Morgan, Freelance Writer,  4/6/2021
How to Submit a Column to InformationWeek
InformationWeek Staff 4/9/2021
White Papers
Register for InformationWeek Newsletters
The State of Cloud Computing - Fall 2020
The State of Cloud Computing - Fall 2020
Download this report to compare how cloud usage and spending patterns have changed in 2020, and how respondents think they'll evolve over the next two years.
Current Issue
Successful Strategies for Digital Transformation
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Flash Poll