Data Integration is the Key to the Connected Future

Satya Nadella has once again told us all where the puck is going to be and invited everyone to skate there with him. At this year’s Microsoft Ignite conference keynote, “A cloud built for the next decade,” Nadella laid out the first compelling vision of a truly distributed computing future, prominently putting Microsoft and Azure on the line to go build it.

Data-driven enterprises will become connected enterprises, ones that unleash productivity, lower risk/cost and decrease time to insight/value. Nadella’s keynote was intended to sketch out a broad vision quickly, focusing on a few key points. Two of these are vital, not just to Microsoft’s vision but to all of us: ubiquitous and decentralized computing and sovereign data and ambient intelligence.

The key to both future cloud features is innovative data integration. How we connect data in both the current and future cloud is key to accomplishing Nadella’s digital transformation revolution.

Data Integration is Lagging Behind

The digital world we live in is dominated by the hybrid multi-cloud, as 85% of businesses have data assets in more than one public cloud according to a 2018 IBM study. This data spread creates a major challenge to conventional data integration strategies.

The most salient feature of conventional data integration is it leverages data location in the storage layer of the modern IT stack to integrate data. The result is enterprises spend lots of resources moving data to computation, creating negative, unintended consequences such as semantic drift, uncertainty, and inefficiency. A recent IDC survey predicted 59 zettabytes of data would be created globally in 2021 with a staggering 90% of that data being replicated or copied.

Data integration strategy has not appreciably changed in the past 30 years. Just like the world before hybrid multi-cloud or the rise of the Internet, most data management strategy works by moving and copying data. That’s the one thing that Snowflake, lakehouses, data lakes, Hadoop clusters, pre-cloud data warehouses and conventional databases all have in common. Before we integrate, manage or query data, we move it into or between one or more of these systems. As Nadella said in his keynote, “…we are at peak centralization right now.”

The Future is Distributed

In the distributed world that Nadella describes, decentralization is the next evolutionary step as relying on data replication is no longer realistic, especially given proliferating IT environments. Data growth is not going to slow down, and network performance isn’t going to radically speed up. Fundamentally rethinking enterprise data integration strategy is the only viable option and is consistent with Nadella’s vision when he refers to computing becoming both ubiquitous and distributed.

First, it’s just more efficient, physically, to distribute computing power than it is to replicate an ever-growing data volume over declining or flat performance of computer networks. It’s not a simple problem to solve, but it is the right approach to meet future trends.

Second, Nadella predicts one of the drivers of distributed cloud computing is the growth of data volume, velocity, and variety. The data monster will not be contained by centralized computing confinement strategies. He mentions federated models, multi-modal data models (presumably as a reference to Azure’s Cosmos DB) and the need for data sovereignty -- the right of data owners to control their data.

Data Integration Must Change for This Future to Exist

Nadella’s keynote is visionary, bold and cements Azure’s position as every bit the equal of AWS in terms of cloud futures. Looking at the data management space broadly, recent developments that are congruent with Nadella’s vision, such as connecting data based on its business meaning, irrespective of storage location versus leveraging data located in the storage layer.

Achieving Nadella’s vision of the future will require changes to cloud infrastructure and business models based upon centralization. More critically, it will also require an enterprise data integration strategy that can bring about a distributed future full of promise and potential.

Kendall Clark is founder and CEO of Stardog, the leading Enterprise Knowledge Graph (EKG) platform provider. For more information visit www.stardog.com or follow them @StardogHQ.