Data Governance is Broken

Over the past several years, data governance has emerged as more than just a trendy buzzword. With the passage of GDPR, CCPA, and other industry, government, and healthcare compliance measures, data governance has become a corporate necessity. Yet many chief data officers cite data governance as a major hurdle for their organization.

So how did we get here? Here are the three main reasons why data governance is failing us:

1. A manual approach is no longer practical.

While we’ve made great advancements in areas such as self-service analytics, cloud computing, and data visualization, we’re not there yet when it comes to governance. Many companies continue to enforce data governance through manual, outdated, and ad hoc tooling. Data teams spend days manually vetting reports, setting up custom rules, and comparing numbers side by side. As the amount of data sources increase and tech stacks become more complex, this approach is neither scalable nor efficient.

While data catalogs often market themselves as the answer to data governance, many data leaders find their catalogs lacking in even the most rudimentary aspects when it comes to manual requirements (for example, lineage tracing and data quality checks).

In some organizations, these operations take up a significant amount of time in manually mapping upstream and downstream dependencies -- not to mention the maintenance work required to keep this up to date. Instead, companies should turn to ML and automation to reduce manual work done by data teams. Our advice: let ML do all the heavy lifting and let your team to focus on the things only they can do.

2. Data is ubiquitous; data governance is not.

Data is everywhere and everyone wants to use it. Teams across the enterprise are voraciously collecting and consuming data to make smarter business decisions. As a result, companies are hiring data engineers and data analysts by the bushel, creating additional data assets and pipelines. Analytics that could only be pulled on a weekly basis are now accessible by the hour.

For many companies, increasing the speed of data innovation is critical to survival. While data infrastructure and business intelligence tools have advanced to support this innovation over the past several years, DataOps has lagged behind with most DataOps solutions like data quality alerts and lineage tracking being manual, one-dimensional, and unscalable.

One of the ways in which DataOps and solutions can catch up is by drawing on concepts from software engineering. Many of the problems we face in data are in fact problems that have been solved in engineering, security, and other industries.

3. Data privacy and security are front and center. For everyone. 

What do Uber, Marriott, Facebook, and Equifax have in common? All four companies have recently been the subject of massive data breaches, costing them millions in SEC settlement fees and eroding valuable customer trust.

While very public companies have faced the most scrutiny as a result of data breaches, not even tornado sirens are immune; increasing regulations and media attention to hacks and breaches are making data privacy and security a priority for everyone. Companies big and small need to take these concerns seriously through a robust data governance strategy.

Instead of keeping these conversations among a small and siloed team of experts, CTOs, CDOs, and VPs should put together a cross-functional data committee that sets security and privacy KPIs and keeps the entire organization accountable to meeting them.

Make way for data governance 2.0

Automation/ML, next-gen DataOps, and data privacy and security are not only foundational to innovation, but critical to the future of data governance. Data governance 2.0 will emerge from the convergence of these three trends, taking center stage for not just CDOs but entire organizations.

If you’re struggling with data governance, know that you are not alone. While there’s plenty of room for improvement, we’re excited to see which new approaches rise to meet this challenge. As data leaders, it’s on us to drive home the importance of this topic, both in our companies and the community at large.

After all, if data governance is a top-level concern, it’s about time we treat it like one.

Barr Moses is CEO & Co-Founder of Monte Carlo, a data observability company. Previously, she was VP of customer operations at Gainsight. Prior to that, she was a management consultant at Bain & Company and a research assistant at the Statistics Department at Stanford. She also served in the Israeli Air Force. Barr graduated from Stanford with a B.Sc. in Mathematical and Computational Science.

Keyur Desai is a global enterprise data management, data monetization and analytics executive. He was most recently Chief Data Officer for TD Ameritrade, where he led a globally dispersed team with full budget responsibility.