Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.
Unified Data: The Missing Piece to the AI Puzzle
Unifying data across the organization’s IT environments presents several challenges, with the most significant being data silos.
4 Min Read
Brian Jackson via Alamy Stock
While data-driven AI models are gaining popularity, their success hinges on high-quality underlying data.
A unified data architecture is crucial for creating a holistic view of business operations and avoiding the ramifications of flawed AI. By bringing together disparate data from across the business, a data architecture ensures data context is kept intact, providing a picture of how the data was generated, where it resides, when it was created, and who it relates to.
A strategy that incorporates a data architecture empowers users to access and use data in real time, creating a single source of truth for decision making, and automating data management processes. Further, a unified data architecture will ensure AI models are thoroughly trained with the appropriate business context, to provide accurate, reliable and high-quality output.
“To integrate disparate data sources successfully, we begin by conducting a comprehensive data audit to identify all existing data repositories across the organization,” says Vagner Strapasson, tech lead data engineer at Indicium.
This involves engaging with key stakeholders from various departments to map out their data landscapes, which helps the company understand the types of data available and their locations.
Next comes an assessment of the quality and relevance of the identified data, using profiling tools to detect inconsistencies, duplicates, and other issues.
“Following this, we establish a centralized data governance framework that defines policies, roles, and standards to ensure consistent data management practices across the organization,” Strapasson says.
To ensure successful integration, the company promotes regular communication and collaboration among teams, and continuously monitors and maintains the integrated data to address any issues that arise.
Data Fingerprinting, Unified Strategies
“Modern data architectures emphasize data organization and fingerprinting for efficient access and AI model training,” Sunil Senan, head of data, analytics, and AI at Infosys, explains in an email interview.
This enables access control, synthetic data versioning, and data security measures while accelerating AI development and improving model accuracy and reliability.
On the other hand, dispersed cloud platforms also make centralized data security solutions difficult, increasing cyber security risk and increasing compliance costs. “AI-powered multi-cloud solutions accelerate and scale the multi-cloud journey of an enterprise and enable organizations to harness their own data estates more quickly,” Senan says.
A unified data strategy provides a clear roadmap to manage and govern data on all the required data capability components, envisioning it through business value relevance and ensuring it through a defined funding model to implement and run it in a sustainable manner.
Data strategy roadmaps can help to comprehensively address aspects of people (operating model), process (data capability processes) and technology (data and technology architecture), thereby making it a foundation for building successful AI initiatives. “Measuring data unification success hinges on AI outcomes and business goals,” Senan says.
Clean, Trusted, Governed Data
From the perspective of Gerard Francis, head of product and platforms for data and analytics at JPMorgan Chase & Co., clean, discoverable, and understandable data is essential for scaling AI initiatives. “Data must be registered for easy access, governed, and supported by tools like data catalogs and automated quality checks,” he says in an email interview.
Effective data architecture simplifies the data estate, enables integration with proper controls, and enhances process efficiency and effectiveness while reducing the effort in developing AI models.
“A unified data strategy can significantly reduce the time data scientists spend on accessing, re-formatting, or creating data, thereby improving their effectiveness in developing AI models,” Francis says.
Yaad Oren, managing director of SAP Labs US and global head of SAP BTP innovation, explains that incorporating AI across an organization is not possible without trusted and governed data. “A unified data strategy simplifies the data landscape, maintains data context and ensures accurate training of AI models,” he says.
This leads to more effective AI deployments and allows customers to harness data to drive deeper insights, faster growth, and more efficiency. “A unified date architecture is crucial for creating a holistic view of business operations and avoiding the ramifications of flawed AI,” he adds.
By bringing together disparate data from across the business, a data architecture ensures data context is kept intact, providing a picture of how the data was generated, where it resides, when it was created, and who it relates to. “A strategy that incorporates a data architecture empowers users to access and use data in real time, creating a single source of truth for decision making, and automating data management processes,” Oren explains.
Data Mesh, Data Literacy Strategies
Strapasson says his company has also employed several strategies and tools to consolidate their data assets.
One such strategy has been the adoption of a data mesh architecture, allowing individual departments to manage their data while ensuring its accessibility organization-wide. “This decentralized approach has proven beneficial, especially for generative AI applications that require diverse data,” he says.
The company also implemented a hybrid data storage and processing system, combining the strengths of data lakes and data warehouses. “This versatility enables us to handle various data types, which is crucial for many AI applications,” Strapasson says.
He adds that fostering a culture of data literacy across the organization has led to broader awareness of the significance of data quality, which has enhanced data consolidation efforts. “These strategies and tools have enabled us to build a cohesive, effective data environment that supports our initiatives and drives better business outcomes,” he says.
About the Author
You May Also Like
More Insights
