News
2/19/2004
07:43 PM
Connect Directly
Twitter
RSS
E-Mail

Accuracy and Precision

Analytic accuracy and precision will make or break real-time decision-support systems.



With growing demand for real-time decision support, analytic accuracy and precision are more important than ever. Systems are increasingly embedding analytics and eliminating immediate human oversight from operational processes in the name of speed, efficiency, and economy. They monitor and respond directly to dynamic business conditions, relying on automated measurement, classification, prediction, and execution. Highly automated systems typically do not deal well with incertitude so they had better work correctly — accurately and precisely — from measurement to action.

Techniques to improve accuracy and precision are more widely understood than applied, perhaps because they're often proposed out of sensible context, sold as ends in themselves rather than as one means toward a goal that is influenced by many factors. And when they are applied, they seem to be perceived as magic bullets that on their own, in isolation, will target and solve business problems. Meanwhile the race toward profitability through automation and disintermediation continues, only increasing needs.

With the hope of providing useful perspective, I'll devote this column to discussing a number of accuracy and precision techniques. There's no magic bullet, however, because context and actual requirements are key. We'll start with data quality.

Garbage In

You know the old saw, "garbage in, garbage out." The implication is that data quality is an absolute, a "must have," an end in itself without regard for actual needs that can be met by realistic but limited steps.

Take a hypothetical direct-marketing firm, where nine addresses out of 100 are undeliverable and three are undetected duplicates due to variant spellings of names or addresses. These errors could be tolerable if the cost of correcting them is greater than the estimated value of the 9 percent missed opportunity and the cost of delivering three percent duplicates. The cost of absolute quality may be higher than the return. Spam e-mail is at one extreme of the spectrum, where the cost of sending an email message is so low that a spammer can send to a dictionary's worth of addresses at a known domain, with absolutely no discretion in targeting, in the hope of a small number of sales.

The U.S. population census is at the other extreme: The government interprets the Constitutional mandate to perform an enumeration of the population to mean that it must make a significant effort to count every individual without recourse to statistical adjustment for missed or duplicated individuals.

Because the government cannot meet its accuracy goals through statistical techniques, it is making a huge effort to improve its TIGER geographic database and its Master Address File in order to target survey forms more precisely. But responses are not verified and may nonetheless be nonfactual. For instance, I might identify myself as Aleut and the output tables will dutifully relay my self-reported but incorrect ethnicity. There's no formal data quality problem here, nor is there when someone gives fictitious personal information when signing up to access a Web site. Data quality is important — but only to the extent that efforts don't overshoot the required precision. It is insufficient to ensure accuracy in the face of data issues not related to quality.

Processes and Models

The government conducts an accuracy and coverage evaluation of the census, an independent survey to assess the source and extent of error, but judged that statistically adjusting 2000 census results to improve accuracy would likely introduce errors greater than those adjustments would eliminate. Fixing results may not improve their accuracy because of limitations in the sensitivity of the measurement instrument and of the correction techniques, reinforcing the importance of designing and building accuracy into processing systems. Do a job right, and you won't have to correct the results.

The twin trends of business performance and process management are positive steps given the central role they assign to process manageability and measurability. Both entail modeling organizational dynamics with built-in assessment and decision points and to accommodate evaluation of alternative scenarios. They differ in that one emphasizes process-quality measurement, monitoring, and optimization and the other testable, repeatable, intentional (rather than haphazard) processes.

The long-established software-testing concepts of verification and validation (V&V) come into play on this larger, organizational scale. Verification seeks to show that a model or algorithm is implemented correctly while validation is the determination that you've chosen the right model and algorithms for the problem at hand. For example, in the past, I've picked on analytic software that provides only linear data fitting, which is useless if you're trying to detect and predict periodic effects like seasonality. The programming might be verified as completely corrected but the results will be wildly inaccurate due to the inadequacy of the model applied. Organizational process models need V&V just as much as software programs to ensure that models are apt and well coded. Good-quality modeling, with the added dimension that process models should adapt to changing circumstances, is important for quality results.



Silk Purses Out of Sows' Ears

Let's suppose that you're doing everything right, that you have data quality adequate for your needs and suitable, managed methods, algorithms, and implementations. There may be additional steps you can take to improve accuracy.

I attended a very interesting talk on "Information Awareness: A Prospective Technical Assessment," presented by David Jensen last August at a conference of SIGKDD, the Association for Computing Machinery's Special Interest Group on Knowledge Discovery and Data Mining. Jensen and coauthors Matthew Rattigan and Hannah Blau ("Information awareness: A prospective technical assessment," kdl.cs.umass.edu/papers/jensen-et-alkdd2003.pdf) all from the computer-science department at the University of Massachusetts, put forth the thesis that the inaccuracy of national-security programs such as CAPPS-II — an airline-passenger screening program that would score air travelers according to a statistical model designed to profile potential terrorists — can be reduced by reframing modeling assumptions and reworking analytic processes. Their prescriptions can surely be applied to many different kind of problems. In this particular case, a rate of just a fraction of 1 percent false positives would mean many thousands of erroneously flagged travelers, gumming up security procedures and reducing the odds of catching actual bad guys.

The authors would start by assuming "relational" data sources "providing connections between individual data records" rather than "propositional" sources "in which each instance is characterized by a set of simple propositions that is, age=32, gender=male)" where "each individual is assumed to be statistically independent of any other." That is, expect and exploit statistical patterns in your data. Such patterns — segments and clusters — may be formed by correlations over time, in demographic or other characteristics, or in outcome.

Secondly, the authors believe that using rankings rather than binary classifiers (such as decision variables), where the outputs are scores rather than overly simple true or false values, can also significantly reduce the error rates and boost accuracy. I interpret this prescription as also calling for weighting the reliability of contributing factors in creating derived indicators on which decisions are based. That is, move away from deterministic models that depict a complex world without nuance, in black and white.

Lastly, they advocate use of multipass rather than single-pass inference. Anything worth doing is worth doing twice, so check your work, and do it differently the second (and third) time. You may tease additional information out of a system that refines results and therefore improves their accuracy.

Lessons

Accuracy doesn't happen by itself, rather it's an element to build into operations in the name of quality, tempered by the precision required to achieve enterprise goals. Techniques should not be viewed as an end in themselves — data quality, for instance, is not an absolute — but more appropriately as factors in a larger, overall picture. That dynamism of the big picture dictates managed models that accommodate changing conditions without degradation in accuracy, models that may internalize checks and balances. The accuracy effort will pay off through support for automated decision-making, which is a growing enterprise imperative, like it or not.

Seth Grimes, [[email protected]] is a principal of Alta Plana Corp., a Washington, D.C.-based consultancy specializing in large-scale analytic computing systems.


Accuracy vs. Precision

The terms accuracy and precision are sometimes carelessly used interchangeably. Accuracy describes exactness while precision refers instead to the sensitivity of the measurement, estimate, or computation. If the odometer in my car records 101.2 miles after I've driven only 100.0, its precision is nonetheless in tenths of a mile even while it is about 1 percent inaccurate, with a sensitivity an order of magnitude less than its precision.

An exchange between Captain Kirk and first officer Spock in the Star Trek episode "Errand of Mercy" illustrates the confusion:

Kirk: What would you say the odds are on our getting out of here?

Spock: It is difficult to be precise, Captain. I should say approximately 7,824.7 to one.

Kirk: Difficult to be precise? 7,824 to one?

Spock: 7,824.7 to one.

Kirk: That's a pretty close approximation.

Spock: I endeavor to be accurate.

Kirk: You do quite well.

Although precise to five places, Spock's approximation was really quite inaccurate since the pair did get out of that particular scrape. Spock could perhaps have done better computing the odds from prior probabilities, using Bayesian statistics, given Captain Kirk's past stellar performance in tight spots.


Research Query

I'm preparing a research study on integration of text mining with traditional techniques for analyzing numeric data. Text mining promises to unlock useful, usable knowledge that is hidden away in unstructured documents. It may prove especially powerful when linked with data mining and BI techniques. If you're doing this sort of integrated analytics, thinking about it, or just plain interested in the subject, drop me a line. I'd like to hear your thoughts.


Resources

Additional Columns at IntelligentEnterprise.com:

"The Word on Text Mining," Dec. 10, 2003:

"Futures Shock," Oct. 10, 2003:

Visit the Business Intelligence InfoCenter at www.intelligententerprise.com/info_centers/bi/

Check out Intelligent Enterprise's Playbook "Complying with Sarbanes-Oxley":
With their integrity on the line, CEOs, CFOs, and other corporate officer mean business when they demand "confidence in the numbers." but where can top executives and IT managers turn for answers?

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Email This  | 
Print  | 
RSS
More Insights
Copyright © 2021 UBM Electronics, A UBM company, All rights reserved. Privacy Policy | Terms of Service