Data Outliers: 10 Ways To Prevent Big Data Damage

An Anomaly Should Be Investigated

If the outlier is an anomaly (i.e. an unlikely but genuine piece of data, rather than a mistake) removing it as a matter of course may be unwise. Sometimes anomalies indicate the beginning of a future trend or something else that should be investigated. For example, some diseases are rare or non-existent in some parts of the world, but an isolated case or a cluster of cases may nevertheless appear as the result of a single person's exposure.

"You shouldn’t blindly assume that outliers are errors. Sometimes they are what you are looking for. For example, in fraud detection or cyber-security applications, outliers, or anomalies might signal undesirable activity, and are themselves of interest," said Vadim Bichutskiy, director of data science at data analytics and technology solutions consulting company Innovizo, in an interview.

When an outlier is an anomaly, rather than the result of a mistake, it should be investigated.

(Image: geralt via Pixabay)