Time to Modernize Your Data Integration Framework - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Data Management
Commentary
4/19/2021
07:00 AM
William McKnight, President, McKnight Consulting Group
William McKnight, President, McKnight Consulting Group
Commentary
50%
50%

Time to Modernize Your Data Integration Framework

Data is constantly making its way to new platforms, applications, algorithms, and users. The need for an effective data integration framework is at an all-time high. Are you ready?

Your enterprise, no matter the industry or its scale, is working to leverage data to achieve its strategic objectives. As an IT leader, you and your team need to ensure the business will not be hamstrung, or even worse, tripped up, by limited data access capabilities.

To be able to leverage data, it must be efficiently accessed, combined, governed, and managed. New sources of data, or new data in current sources, must also be found, understood, integrated, and managed.

Credit: gerasimov174 via Adobe Stock
Credit: gerasimov174 via Adobe Stock

While the pain caused by a lack of mature data integration may be related to people or process, you must also ask if your enterprise has adopted the right product set or if it is stuck in outdated tools. The technical debt that has accumulated from years of workarounds and gap-fixing existing processes may seem too expensive to simply rip and replace with more capable modern tools. However, the demand for mature, modern data integration is becoming too strong to ignore. Getting by with an outdated platform, or not using a modern platform to its fullest, is no longer a sustainable choice.

We’ve seen the evolution to truly data-driven organizations through digital transformation. Now, we see the latest evolution where mature enterprises are leveraging artificial intelligence (AI) and machine learning, powering data integration to automate tasks and guide the user experience. You want this.

Although the latest evolutionary stage -- and the new high watermark -- of data integration is AI-powered automation and enablement, there are more requirements such as cloud-native deployments and enterprise scale and trust.

You need to be able to orchestrate the ebb and flow of data among multiple nodes, either as multiple sources, multiple targets, or multiple intermediate aggregation points.

The data integration platform must also be cloud native today. This means the integration capabilities are built on a platform stack that is designed and optimized for cloud deployments and implementation. This is crucial for scale and agility -- a clear advantage the cloud gives over on-premises deployments.

Additionally, data management centers around trust. Trust is created through transparency and understanding, and modern data integration platforms give organizations holistic views of their enterprise data and deep, thorough lineage paths to show how critical data traces back to a trusted, primary source.

Finally, we see modern data analytic platforms in the cloud able to dynamically, and even automatically, scale to meet the increasing complexity and concurrency demands of the query executions involved in data integration. The new generation of some data integration platforms also work at any scale, executing massive numbers of data pipelines that feed and govern the insatiable appetite for data in the analytic platforms.

The “Egregious Toil and Labor” of conventional ETL, where development and change takes months, must become an approach of the past. Intelligently driven automation suggests and generates new data pipelines between source and target without manually mapping or design, saving and optimizing steps. Here are the critical capability categories which define the capabilities for data integration competitive advantage:

  • Comprehensive Native Connectivity
  • Multi-Latency Data Ingestion
  • Data Integration (in all of: ETL, ELT, Streaming)
  • Data Quality and Data Governance
  • Data Cataloging and Metadata Management
  • Enterprise Trust at Scale
  • Artificial Intelligence and Automation
  • Ecosystem and Multi-Cloud

Data integration has always been the most important component in leveraging data to achieve enterprise strategic objectives. It is evolving with artificial intelligence and other critical capabilities. The opportunity exists to truly make a difference in not just the data architecture, but also the enterprise, by leveraging these capabilities.

Related Content:

Enterprise Analytics Kicks Off to a Promising Start

IT Disappoints Business on Data and Analytics

5 Tips for Understanding and Maximizing Data

William McKnight has advised many of the world's best-known organizations. His strategies form the information management plan for leading companies in various industries. He is a prolific author and a popular keynote speaker and trainer. He has performed dozens of benchmarks on leading database, data lake, streaming and data integration products. William is a leading global influencer in data warehousing and master data management and he leads McKnight Consulting Group, which has twice placed on the Inc. 5000 list. He can be reached at [email protected].

The InformationWeek community brings together IT practitioners and industry experts with IT advice, education, and opinions. We strive to highlight technology executives and subject matter experts and use their knowledge and experiences to help our audience of IT ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Slideshows
10 Things Your Artificial Intelligence Initiative Needs to Succeed
Lisa Morgan, Freelance Writer,  4/20/2021
News
Tech Spending Climbs as Digital Business Initiatives Grow
Jessica Davis, Senior Editor, Enterprise Apps,  4/22/2021
Commentary
Optimizing the CIO and CFO Relationship
Mary E. Shacklett, Technology commentator and President of Transworld Data,  4/13/2021
White Papers
Register for InformationWeek Newsletters
2021 State of ITOps and SecOps Report
2021 State of ITOps and SecOps Report
This new report from InformationWeek explores what we've learned over the past year, critical trends around ITOps and SecOps, and where leaders are focusing their time and efforts to support a growing digital economy. Download it today!
Video
Current Issue
Planning Your Digital Transformation Roadmap
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Slideshows
Flash Poll