Avoid Bad-Data Potholes - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Software // Enterprise Applications
News
6/13/2003
03:33 PM
50%
50%

Avoid Bad-Data Potholes

In this day of stepped-up data analysis, data quality is a critical issue that needs more attention

Nobody had to convince Jim Eardley, managing director of business development and strategy for FleetBoston Financial, that data quality is important. FleetBoston's highest-value customers have multiple accounts with the bank and, therefore, multiple touch points. To meet their needs effectively, Eardley says, "everyone must be reading from the same script, and that script has to be exactly right."

But achieving and maintaining clean data isn't easy. The rapid growth of E-commerce, the ongoing integration of multiple databases, and the rise in the use of business intelligence across companies have only amplified the problem. The more information companies access, the more unreliable it is. "We've gone from IT environments where there were a few data-entry clerks, to a world where everyone is a data-entry clerk," says Mike Schiff, an analyst at Current Analysis.

Data cleansing in the past has been done as a batch job, comparing name-and-address records with available reference material, or on a case-by-case basis when a customer complains about faulty information. "Most companies have seen data quality as a tactical IT problem rather than a strategic advantage for business," says Ted Friedman, a Gartner analyst. "That's just the wrong way to do it. But most companies don't have data quality on their radar yet."

They should, given that the cost of erroneous mailings--including postage, printing, and staffing--hit $611 billion for U.S. businesses in 2002, according to a study by the Data Warehouse Institute. And that doesn't include a potentially bigger loss: missed opportunities because of bad marketing decisions that are based on faulty data.

Rather than looking at maintaining the integrity of data as a necessary evil, companies should see data quality as a critical business process, and clean data a key product of the business. "You need to build your database around your constituents. And then you need to have a corporate commitment to maintain the quality of that data," says Ron Boeving, VP of information services and CIO at First Health Group Corp., a managed-care service provider that uses data-cleansing software from Group 1 Software. "If we don't control the quality of our data, we have no products," he says.


Andy Lesser, a senior technical analyst at FedEx Corp. Photo by Rochelle Mozeman.

One of the hardest parts of data reengineering is deciding what "bad data" is and what your company will accept, says Lesser, a senior technical analyst at FedEx.
But doing that isn't easy. Part of the problem is that databases in a company of any size are typically owned by individual lines of business. Each database administrator manages the data to a sufficient level of quality for the department. For example, a call center may allow operators to input addresses without apartment numbers, while a service manager wouldn't. "In data reengineering, one of the hardest problems is defining what bad is," says Andy Lesser, a senior technical analyst at FedEx Corp.

Databases created at different times, in different formats, and for different business purposes may look similar, but undocumented assumptions about fields can lead to errors even in hand-checked records. When First Health was importing data from a company that it recently acquired, it thought the data was relatively clean. But then the undocumented data started to create problems. "We saw 'start date'" as a field in the database, says Bob Bularzik, assistant VP of software technologies at First Health. "But the start date of what?"

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Previous
1 of 2
Next
Comment  | 
Print  | 
More Insights
Commentary
Enterprise Guide to Digital Transformation
Cathleen Gagne, Managing Editor, InformationWeek,  8/13/2019
Slideshows
IT Careers: How to Get a Job as a Site Reliability Engineer
Cynthia Harvey, Freelance Journalist, InformationWeek,  7/31/2019
Commentary
AI Ethics Guidelines Every CIO Should Read
Guest Commentary, Guest Commentary,  8/7/2019
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Data Science and AI in the Fast Lane
This IT Trend Report will help you gain insight into how quickly and dramatically data science is influencing how enterprises are managed and where they will derive business success. Read the report today!
Slideshows
Flash Poll