Software // Information Management
Commentary
7/28/2010
01:58 PM
Curt Monash
Curt Monash
Commentary
Connect Directly
RSS
E-Mail
50%
50%

What To Do About False-Positive Alerts, Non-Collaborative BI and Inaccurate Metrics

"Push" alerting technology could be much more granular and useful, but is being held back by the problem of false positives. Metrics passed down from on high didn't work too well in Stalin's USSR, and haven't improved sufficiently since. A piece of the solution to both problems is an engine for setting and modifying metrics definitions.

I've been hinting at some points for quite a long time, without really spelling them out in written form. So let's fix that. I believe:

  • "Push" alerting technology could be much more granular and useful, but is being held back by the problem of false positives.
  • Metrics passed down from on high didn't work too well in Stalin's USSR, and haven't improved sufficiently since.
  • A large, necessary piece of the solution to both problems is a great engine for setting and modifying metrics definitions.
I shall explain.False positives on alarm systems are a huge problem. Alarms were ignored (indeed, turned off) on the Deepwater Horizon due to too many being false. When I had a fire in my house, the smoke alarm wasn't on because of its penchant for earsplitting false positives. And false positives held back the software category of intrusion detection systems until it was eventually subsumed by related kinds of security appliance. Similarly, false positives are an under-appreciated problem that restrains the advance of business intelligence software. It's no big inconvenience when, out of your 3 biggest problems this week, your dashboard calls attention to 17 of them. But suppose you want to make alerts more granular, take them up in quantity a couple of orders of magnitude, and have your smartphone pinged each time the alert condition is met. Whoa. Now those bogus warnings start to be more of a consideration. Why would you want some kind of "push" alerting? Use case circumstances include but are not limited to anything that might herald:
  • A machine malfunctioning or reaching capacity limits (many kinds of machine).
  • A critical shortage (inventory or supply).
  • A sudden market trend change (if you're in the kind of consumer market where significant developments an occur in hours or less)
  • An investment opportunity or threat, or a financial risk threshold being breached (if you're an investor, if you buy or sell commodities, or if you're a corporate treasurer).
  • An attitude change on the part of a specific important customer (sales contact, service contact, news development, whatever).
  • The advisability of rerouting or rescheduling something.
Why would these alerts ideally be granular and high-volume? Well, you might have lots of machines, lots of trucks, lots of customers, lots of potential investments, and so on. I could list many more examples, but these probably suffice to at least start the discussion.

I'm agnostic about what the UI would be. Sounds going off on your mobile phone? Email on your mobile device? An app more like a chat window? Something more like a mobile dashboard? Whatever it is, false positives would really screw up your work day.

Classical enterprise dashboards suffer from a similarly fundamental problem -- dashboard technology is optimized for a screwed-up enterprise analytic methodology. That is:

  • Somebody tells you what you're supposed to care about.
  • Somebody tells you how to measure what you're supposed to care about.
  • The simpler and more top-down all this is, the better it creates enterprise-wide unity of vision/focus/purpose/blahblahblah.
Yes, you can do drilldown or even data exploration, QlikView-style or otherwise. You can communicate around the BI tools, whether via general portal technology or something built-in. Vendors keep trying to make it easier for you to build your own reports, charts, and dashboard elements. But you can't very easily define and track your own metrics, and I don't see a lot of effort toward making that better.

Perhaps vendors don't try to provide strong metrics management because doing so might lead to -- gasp! -- more than "one version of the truth." If so, such thinking is regrettably misguided. Examples of formulas -- i.e., metrics -- that should and can not be cast in stone* include:

  • Profitability of a customer
  • Expected lifetime value of a customer
  • Anything else which boils down to "importance of a customer"
  • Profitability, value, or importance of a lead or prospect action
  • Significance of a social media flare-up
  • Cost of a schedule slip
  • Risk of a portfolio
*At least as a general rule -- we surely can think of some simple cases or exceptions where extreme cookie-cutterness is just fine.

It's not that there shouldn't be preferred or default metrics for these quantities. Rather, my point is that individuals should be allowed to:

  • Question those metrics
  • Consider and test alternatives to those metrics
  • Make decisions based on those alternatives (as long as they also keep the results and implications of the approved/default versions of the metrics in mind)
Why? Because the person closest to the situation may be the one best able to assess it, and also because recognizing that fact is a really good idea from the standpoints of employee morale, engagement, and even development. (Or you might say -- shiny buzzword! -- employee "empowerment".) In particular, when a boss and her subordinate disagree about the ideal way to calculate a metric, they should be able to track both versions side-by-side. When there's a significant difference between the two, that could be a fine time for an out-of-the-norm decision.* If that ability were available, portal-style BI collaboration would almost automatically come into its own.

*What kind of decision that is of course depends on the specifics of the case. If only one of two competing metrics says you should bother giving extra care to a specific customer, the answer is easy - you give it. In other cases, an actual meeting or other conversation may be in order to decide what to do.

A metric is just a function. To offer BI users the flexibility I want, it should be very easy to use these functions as inputs into other, custom functions. I.e., I'm talking about something with the flexibility of a stripped down Excel, although the UI would be more like that for a rules engine. And if you think about it, that's exactly the same functionality needed to personalize your alerts and weed out false positives.

One last point -- what I'm calling for could greatly increase the BI performance burden, perhaps even by three orders of magnitude. Well, that's what we have all this great new super price-performance analytic database technology for. And anyhow the throughput hit won't come overnight. It will be interesting to see where the boundaries end up between DBMS and inside-the-BI-tool analytic processing. But overall, the analytic DBMS industry -- and the hardware vendors backing them up -- should be able to handle anything the BI vendors throw at them.

Related links

"Push" alerting technology could be much more granular and useful, but is being held back by the problem of false positives. Metrics passed down from on high didn't work too well in Stalin's USSR, and haven't improved sufficiently since. A piece of the solution to both problems is an engine for setting and modifying metrics definitions.

Comment  | 
Print  | 
More Insights
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest - August 20, 2014
CIOs need people who know the ins and outs of cloud software stacks and security, and, most of all, can break through cultural resistance.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.