Data Silos: Now and Forever?

Here's something most IT and business leaders can agree on: Data silos are bad and should be eradicated. Unfortunately, after decades of criticism and proposed solutions, silos remain stubbornly intact at all too many organizations, impeding the free flow of data and strangling analytics.

Brian Moffo, director of analytics delivery for IT services provider Anexinet, blames inertia, fear and an overzealous focus on organizational readiness for silos' enduring enterprise presence. "Most silos start out of necessity and have existed for a long period of time," he observed. Silo managers don't always understand the advantages of collaboration. "They often throw up walls and boundaries to access the data of those silos because they feel like they are the only ones who can manage it," Moffo explained.

Organizations that have data in silos have a much harder time employing a holistic management approach in which operations and tasks are optimized across the organization, noted Pedro Ferreira, associate professor of information systems at Carnegie Mellon University's Heinz College of Information Systems and Public Policy. "Silos of data usually prevent organizations from understanding that changing one part of the system affects other, potentially remote parts of their system," he said. "Because the changes in the data do not flow from one dataset to the next, one observes change but is unable to track why and how changes cascade."

The big hurt

John Thielens, CTO of data integration software provider Cleo, said silos hurt enterprises in five specific ways:

1. Slow data-driven decision making, which can affect a company’s ability to efficiently compete in today’s fast-paced economy, one that’s rapidly undergoing digital business transformation.

2. Poor trust and collaboration across various teams within a company.

3. Higher costs due to duplicate and redundant IT and application infrastructure.

4. Reduced credibility and quality of data, which hinders the ability to quickly analyze and leverage the value of data analytics and insights.

5. A poor experience across the end-to-end customer journey.

Thielens noted that many organizations don’t fully comprehend the value of having an integrated IT infrastructure, one in which holistic data can be used to create new ways of generating customer value while simultaneously improving business operations. "Integration tears down these data silos," he added. "It connects the systems and applications that drive critical business workflows so organizations can automate and streamline the order-to-cash cycle and reduce the number of processing errors." The overall result, according to Thielens, is improved visibility into everyday operations and a better experience for customers and business partners.

Getting it together

Kevin Harper, a data scientist at The Intern Group, an organization that provides international internship programs, recently consolidated the data silos based at its global offices. "It can be intimidating to think that you have data isolated in 10 or 20 locations around the world," he observed. "It can be more challenging when the data you have is in different formats."

Kevin Harper

Harper said that The Intern Group decided to achieve silo consolidation as a way to streamline its everyday business tasks. "What we have uncovered along the way are many opportunities to build data science tools that support our operations," he noted. "Without first consolidating our data, we would not have been able to move on to more advanced analytics and data science."

Yet silo consolidation, even under the best of conditions, can be both challenging and time consuming, cautioned Erik Gfesser, principal architect at IT consulting firm SPR Consulting. "This is exactly the case if applications are built directly on top of the data being tackled with little or no abstraction, or business logic is distributed across both the application and data layers," he stated. "If business logic resides in the data layer, consolidation can be especially challenging when attempting to also simultaneously migrate to a new database product, as stored procedures and functions tend to be an area where database products tend to differentiate."

Another challenge is the risk of incorrectly interpreting data during the consolidation process, leading to inconsistencies. "The problem here is that each silo is likely to have at least some data that is seemingly similar, but should really be interpreted differently," Gfesser explained. A simple example is party data comprising either people or organizations. "In some professional domains, an organization name might be represented by an individual," he noted.

John Thielens

Silo consolidation can also be hampered by each department's unique data management preferences and idiosyncrasies. "Habits are hard to change quickly or without a learning process and some pushback," said Jeff Looman, vice president of engineering at FileShadow, a cloud-based data backup service provider.

A successful silo consolidation project begins with getting leadership onboard. "Data migrations can be complicated and long-term projects, so you need to start with leadership to inspire motivation throughout the organization," Harper suggested. "After that, it comes down to finding the right talent to lead the transition."

Jeff Looman

The best way to achieve a smooth and accurate consolidation is to bring in a mediator who's familiar with both silos and can speak to both sides, Looman advised. "The mediator can then recommend a consolidation solution that will work best for all, or the most, parties," he stated.

Big mistakes

Despite growing evidence to the contrary, many organizations continue to stick their collective ostrich-like heads into the sand, assuming that their silos will never have to interact with each other or that some type of quick fix will magically change the dynamic, Looman noted. "Not considering pilot program execution first to see how effective silo merging might be is another key mistake," he added.

Another important error, Harper pointed out, is underestimating the value silo consolidation gives an organization, "While some might see consolidated data as a helpful way to organize information, it is more important that leadership understand the value you unlock by consolidating data," he explained.