Thinking About Analytic Speed - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Software // Information Management
Commentary
9/14/2009
09:36 AM
Curt Monash
Curt Monash
Commentary
50%
50%

Thinking About Analytic Speed

There is, obviously, a huge industry emphasis on speed. Indeed, there's so much emphasis that confusion often ensues... There are two important senses of "latency" in analytics. One is just query response time. The other is the length of the interval between when data is captured and when it is available for analytic purposes. They're often conflated...

For a variety of reasons, I don't plan to post my complete Enzee keynote slide deck soon, if ever. But perhaps one or more of its subjects are worth spinning out in their own blog posts. I'm going to start with analytic speed or, equivalently, analytic latency. There is, obviously, a huge industry emphasis on speed. Indeed, there's so much emphasis that confusion often ensues. My goal in this post is not really to resolve the confusion; that would be ambitious to the max. But I'm at least trying to call attention to it, so that we can all be more careful in our discussions going forward, and perhaps contribute to a framework for those discussions as well.Key points include:

1. There are two important senses of "latency" in analytics. One is just query response time. The other is the length of the interval between when data is captured and when it is available for analytic purposes. They're often conflated - and indeed I shall do so for the remainder of this post.

2. There are many different kinds of analytic speed, which to a large extent can be viewed separately. Major areas include:

3. It is indeed important to carefully assess your need for speed. Acceptable levels of analytic latency vary widely, ranging from sub-millisecond to multi-month. For example, I've put together a list:

  • Algorithmic trading - Sub-millisecond. Increasingly, that's what's needed, at least for query response.
  • Web page - Tenths of seconds. If you want to get up a complex web page in 2 seconds or less, you may require sub-second response time for your queries. (E.g., this is a key message from Teradata's customer success story at Travelocity.)
  • Call center - Seconds. If two humans are talking to each other on the phone, a couple-second delay in response is probably acceptable.
  • Transportation - Tens of minutes. If a commercial flight is delayed, reaction to minimize the consequences often needs to be sub-hour. The same can be true for cargo transportation (truck, rail, or air). In other cases, a couple of hours may be fast enough.
  • Inventory - Hours. In the 1980s, the retailers that won were the ones who reordered hot seasonal merchandise a couple of days before their competitors. Even then, 7-11 Japan was making restocking decisions several times a day. Things have only gotten faster since.
  • Planning - Weeks or more. Planning is often done on an annual or even multi-year cycle. That may be excessively slow. But weeks or months? In many cases, that's both the best achievable and plenty good enough.

That's a range of at least nine orders of magnitude, which is a lot like the difference between the speed of a turtle and the speed of light.There is, obviously, a huge industry emphasis on speed. Indeed, there's so much emphasis that confusion often ensues... There are two important senses of "latency" in analytics. One is just query response time. The other is the length of the interval between when data is captured and when it is available for analytic purposes. They're often conflated...

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
News
IBM Puts Red Hat OpenShift to Work on Sports Data at US Open
Joao-Pierre S. Ruth, Senior Writer,  8/30/2019
Slideshows
IT Careers: 10 Places to Look for Great Developers
Cynthia Harvey, Freelance Journalist, InformationWeek,  9/4/2019
Commentary
Cloud 2.0: A New Era for Public Cloud
Crystal Bedell, Technology Writer,  9/1/2019
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Data Science and AI in the Fast Lane
This IT Trend Report will help you gain insight into how quickly and dramatically data science is influencing how enterprises are managed and where they will derive business success. Read the report today!
Slideshows
Flash Poll