Data Management Heads into Major Transition

One of the in-demand, emerging, and well-paid jobs that has emerged in the IT enterprise over the last decade may soon be getting some long-awaited help.

Data engineers are the IT pros who have been the experts on the front lines of big data and machine learning, getting all the parts and pieces of data, data infrastructure, data management, and data tools to work together. There's no formal training program to be a data engineer in the enterprise. It's really been a MacGyver-type effort performed by talented pros who learn on the job.

But help is on the way, according to Alan Duncan, research vice president at Gartner.

"Very broadly we are on the cusp of a transition from a piecemeal architecture," he said. That means that organizations won't have to buy a range of individual components that the data engineer then must cobble together into a coherent environment that enables enterprise workers to perform business intelligence and advanced analytics. We are moving beyond that kind of early effort toward the next level of maturity -- a data mesh or a data fabric that includes all the needed pieces.

"The providers we are working with are creating a more bundled and packaged view of how those components come together. You can buy a much more integrated approach to the data management environment," Duncan said.

Among the components that are being bundled into platforms today are the automation of data discovery and data provisioning, AI-enabled graph analytics that look at existing data and how it connects to other data, and more.

This transition has been driven by other changes we've already experienced over the last 5 to 10 years. For instance, the growth of data as it became "big data." That term may be less popular now, but the concept still applies. Organizations are collecting more data than they ever have before. Managing and analyzing these large data sets has propelled businesses to new insights and made machine learning more viable in the enterprise. But it's created new challenges as well. Now that platforms are catching up by incorporating a lot of the functions that slowed down data efforts in the enterprise, data management is at a turning point.

But we are just at the beginning, Duncan said. This shift has started in the last 12 months, and it will take another 2-plus years. 

A number of big vendors are working to create a unified platform, including Amazon, IBM, Informatica, and Oracle. Some service providers and systems integrators, including Accenture and Infosys, are also working on building a unified platform for their clients. But the platforms are just one piece of the puzzle, and it's not where the value truly lies, according to Duncan.

"The value is not really in the platforms. They are not the end in themselves," he said. "The value is in the data."

Venders are looking to add that value, too, by acting as a data provider or broker. For instance, IBM's acquisition of The Weather Channel gave the company access to a rich source of weather intelligence, Duncan said. Other vendors are acting as data exchange markets. Vendors can then offer these data sources to their customers, saving enterprises the work of having to forge relationships with multiple providers of data. Gartner is predicting that by 2022, 50% of cloud buying decisions will be based on the data assets provided by cloud service providers rather than on the product capabilities. 

Even as data management is about to enter a new era, the world is dealing with a real crisis. Duncan said that as an optimist, he is hopeful a side effect of the coronavirus pandemic will be to teach the world about data-driven conversations.

"All of these things are happening in a microcosm with respect to COVID-19," Duncan said. "There's been a great appetite from society to collaborate...There's a potential here for us as a society to learn about what being more data literate means, for paying more attention to data and science."

For more on data management, check out these stories:

Analytics Pros Called to Duty in COVID-19 Crisis

Future IT Teams Will Include More Non-Traditional Members

COVID-19: Using Data to Map Infections, Hospital Beds, and More

IT Disappoints Business on Data and Analytics