Data Management Heads into Major Transition - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Data Management // Big Data Analytics
News
4/7/2020
08:30 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Data Management Heads into Major Transition

Data management and the tools that power it are on the cusp of a major transition that is poised to make the jobs of data engineers easier.

One of the in-demand, emerging, and well-paid jobs that has emerged in the IT enterprise over the last decade may soon be getting some long-awaited help.

Data engineers are the IT pros who have been the experts on the front lines of big data and machine learning, getting all the parts and pieces of data, data infrastructure, data management, and data tools to work together. There's no formal training program to be a data engineer in the enterprise. It's really been a MacGyver-type effort performed by talented pros who learn on the job.

Image: WrightStudio - stock.adobe.com
Image: WrightStudio - stock.adobe.com

But help is on the way, according to Alan Duncan, research vice president at Gartner.

"Very broadly we are on the cusp of a transition from a piecemeal architecture," he said. That means that organizations won't have to buy a range of individual components that the data engineer then must cobble together into a coherent environment that enables enterprise workers to perform business intelligence and advanced analytics. We are moving beyond that kind of early effort toward the next level of maturity -- a data mesh or a data fabric that includes all the needed pieces.

"The providers we are working with are creating a more bundled and packaged view of how those components come together. You can buy a much more integrated approach to the data management environment," Duncan said.

Among the components that are being bundled into platforms today are the automation of data discovery and data provisioning, AI-enabled graph analytics that look at existing data and how it connects to other data, and more.

This transition has been driven by other changes we've already experienced over the last 5 to 10 years. For instance, the growth of data as it became "big data." That term may be less popular now, but the concept still applies. Organizations are collecting more data than they ever have before. Managing and analyzing these large data sets has propelled businesses to new insights and made machine learning more viable in the enterprise. But it's created new challenges as well. Now that platforms are catching up by incorporating a lot of the functions that slowed down data efforts in the enterprise, data management is at a turning point.

But we are just at the beginning, Duncan said. This shift has started in the last 12 months, and it will take another 2-plus years. 

A number of big vendors are working to create a unified platform, including Amazon, IBM, Informatica, and Oracle. Some service providers and systems integrators, including Accenture and Infosys, are also working on building a unified platform for their clients. But the platforms are just one piece of the puzzle, and it's not where the value truly lies, according to Duncan.

"The value is not really in the platforms. They are not the end in themselves," he said. "The value is in the data."

Venders are looking to add that value, too, by acting as a data provider or broker. For instance, IBM's acquisition of The Weather Channel gave the company access to a rich source of weather intelligence, Duncan said. Other vendors are acting as data exchange markets. Vendors can then offer these data sources to their customers, saving enterprises the work of having to forge relationships with multiple providers of data. Gartner is predicting that by 2022, 50% of cloud buying decisions will be based on the data assets provided by cloud service providers rather than on the product capabilities. 

Even as data management is about to enter a new era, the world is dealing with a real crisis. Duncan said that as an optimist, he is hopeful a side effect of the coronavirus pandemic will be to teach the world about data-driven conversations.

"All of these things are happening in a microcosm with respect to COVID-19," Duncan said. "There's been a great appetite from society to collaborate...There's a potential here for us as a society to learn about what being more data literate means, for paying more attention to data and science."

For more on data management, check out these stories:

Analytics Pros Called to Duty in COVID-19 Crisis

Future IT Teams Will Include More Non-Traditional Members

COVID-19: Using Data to Map Infections, Hospital Beds, and More

IT Disappoints Business on Data and Analytics

Jessica Davis has spent a career covering the intersection of business and technology at titles including IDG's Infoworld, Ziff Davis Enterprise's eWeek and Channel Insider, and Penton Technology's MSPmentor. She's passionate about the practical use of business intelligence, ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Slideshows
10 Ways to Transition Traditional IT Talent to Cloud Talent
Lisa Morgan, Freelance Writer,  11/23/2020
News
What Comes Next for the COVID-19 Computing Consortium
Joao-Pierre S. Ruth, Senior Writer,  11/24/2020
News
Top 10 Data and Analytics Trends for 2021
Jessica Davis, Senior Editor, Enterprise Apps,  11/13/2020
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Why Chatbots Are So Popular Right Now
In this IT Trend Report, you will learn more about why chatbots are gaining traction within businesses, particularly while a pandemic is impacting the world.
Slideshows
Flash Poll