10 Big Data Migration Mistakes

Transferring data between computer systems or storage formats is never a trivial task, particularly when it involves both structured and unstructured data. The complexity of data-migration jobs means that cost overruns and delays with "go-lives" are all too common, said Arvind Singh, co-founder and CEO of Utopia, a Chicago-based enterprise data solutions provider.

In a phone interview with InformationWeek, Singh outlined 10 common data-migration problems--five pitfalls and five risks--that enterprises should strive to avoid.

Pitfall #1: Failing to engage the lines of business and business users at the outset.
When companies integrate or consolidate multiple systems into one--often after a business merger--they need to identify the right business uses at the outset.

"You need to identify who knows and understands the business data," said Singh. "Who's the subject matter expert in your business? It's certainly not IT or the systems integrator."

In other words, bring the people who'll be using the data into the migration project. After all, they'll be the ones operating the system once it goes live.

2. Absence of data governance policies and organizational structure.
"You've got data being moved from System A to System B, but who owns the governance structure? Who has the rights to create, approve, edit, or remove data from the system?" Singh asked.

[ Now that you have all this data, what do you do with it? Learn 5 Ways To Benefit From Big Data. ]

Other issues that must be resolved: Is your organization set up to manage data? Is there a business process for managing the lifecycle of data? And do you have data stewards in the company?

Pitfall #3: Poor data quality in a legacy system.
Companies often don't realize that an "as-is assessment" is essential before embarking on a data-migration job.

"Understanding the quality of existing data in a legacy system is a huge pitfall that companies often don't spend enough time on," said Singh

Questions to consider: Will the existing data support new users? What is it missing? And what are you planning to do, analysis-wise, that you're not able to do today?

A detailed assessment makes it easier for companies to estimate the amount of work required to migrate legacy data successfully.

Pitfall #4: Neglecting to validate and redefine business rules.
Your company's business and validation rules may not be current.

"It's amazing how little time companies have spent agreeing on a business rule, much less making sure the data complies with the business rule," said Singh. "In other words, you think you have a business rule, but does your existing data match, map, or comply with that rule?"

In addition, auditors need to be sure that data moved from a legacy system to a new system has been validated, especially when a migration involves critical information such as financial, inventory, and payroll data.

Pitfall #5: Failure to validate and test the data-migration process.
Don't save this step for the end. "You really need to make sure that you're validating and testing throughout the process," Singh said.

Questions to consider: How are you going to test the data? Who will test and evaluate it? Who will sign off on it? And who's the ultimate consumer of the data?

This process must be built into the project's lifecycle, but unfortunately companies often "don't spend enough time aligning the data testing, validation, and migration cycles to the project timeline," said Singh.