Real-Time Analytics: 10 Ways To Get It Right
While real-time analytics is getting more affordable, it's still not right for everything. Here are 10 ways to get the most from real time, near real time, and batch use cases.
![](https://eu-images.contentstack.com/v3/assets/blt69509c9116440be8/bltf3072d20d5d2d277/64cb4e82acb8b24d74a17ba6/1-superhero-534120_1280.jpg?width=700&auto=webp&quality=80&disable=upscale)
Real time, near real time, batch. Organizations need actionable insights faster than ever before to stay competitive, reduce risks, meet customer expectations, and capitalize on time-sensitive opportunities.
CIOs have been pummeled with requests for real-time analytics because people in the organization think they need it -- in marketing, IT, security, fraud prevention, customer support, and other areas -- and some of them actually do need it. In the not-so-distant past, very few reasons justified the expense of real-time analytics, but with the cloud, a new generation of solutions, and open source projects like Apache Hadoop and Spark, the economics have changed. As a result, the scope of the use cases is expanding.
Whether to choose real time, near real time, or batch "depends on the use case and how important it is to get an up-to-the-second response. It's all about the response," said John Bates, CMO and former CTO for intelligent business operations and big data at Software AG, in an interview. "Reports that used to be available at the end of the month or in a week are now available intraday, and then you're getting into 5, 10, 15 minutes. That's fine for people who want dashboards, but if you're doing high-frequency trading or trying to stop a security or compliance threat before it causes damage, it's critical to receive the lowest latency response."
While it's clear that the time-to-insights window is collapsing, it's less clear what individuals or companies mean when they talk about real time and near real time, since the definition can vary depending on the need, the industry, and an individual's point of view. Real time is often defined in microseconds, milliseconds, or seconds, and near real time in seconds, minutes, or hours -- although the definitions can vary even more than that. More important than a universal definition of the categories is the business need, viewed in terms of cost and benefits (usually capitalizing on opportunities, minimizing risks, and satisfying customers).
"We talk about 'just in time' to help [customers understand their] cycles and how they perceive that changing, because it goes to the type of investment," said Keith Collins, CIO at analytics and business intelligence (BI) solution provider SAS, in an interview. "Is it the speed of making the decision, or how fast you want to look at your historical basis? Most people aren't going to change their models quickly."
For one thing, there are hurdles to overcome, such as adopting new technologies, making architectural adjustments, and grappling with data integration issues. Accelerating insights may also require business process adjustments, some of which may be met with resistance by users. The progression to real time and near real time is often gradual, meaning there is a speed improvement of one or two orders of magnitude, such as hours to minutes or seconds, and then further refinement as necessary, driven by business need.
"[Companies] understand the need to analyze real-time and historical data together to identify system or application-level patterns as they occur, as well as to generate meaningful business impact and deliver value to their own customer base," said Ankur Goyal, vice president of engineering at MemSQL, in an interview. "Because end users have come to expect short load times, personalization, and updates in real time, it is vital to replace legacy architectures with a real-time data pipeline to capture, process, analyze, and serve massive amounts of data to millions of users."
In today's dynamic business environment, companies must be more nimble than ever. Here's how the time element is playing out in analytics across enterprises and industries.
Interactive TV has to be more than gee-whiz cool. It has to drive ROI for advertisers and provide a compelling viewer experience. When advertisers buy TV spots, they've traditionally depended on metrics such as audience size, audience demographics, and show ratings, which are inadequate by today's standards. Today, advertisers want granular information about their audiences, and the ability to target their efforts more effectively.
"Advertisers are demanding detailed audience data before they sign on the dotted line and commit millions to TV. That goes for Turner, Turner Sports, ESPN, ABC, NBC, Disney, and Univision," said Gavin Douglas, a TV producer and CEO of TV participation company iPowow, in an interview.
Advertisers have always wanted to know who watched the show during which their ads ran. To be relevant in today's media environment, they demand to know who is currently watching the show, so advertising can be adjusted dynamically to suit the audience. By 2017, programmatic advertising will push different ads to different people watching the same show on different TVs, laptops, or mobile devices.
"At that point, real-time knowledge of the audience and real-time targeting is going to be a huge market," Douglas said. "The industry is hitting a stride because it's been forced to [do so] by the digital wave of knowing your audience and real-time interaction on a website with an audience vs. 'we don't care because this is how we make linear TV.'"
Meanwhile, the content side is becoming increasingly interactive and socialized. For example, MTV recently ran a 12-hour "Teen Wolf" marathon that engaged viewers with trivia questions, predictive gaming questions, prizes, and the opportunity to appear on the show. Approximately 160,000 viewers participated, many of whom connected through Facebook, Twitter, or an email account.
"Apart from [long-form drama], every TV genre is expanding in its use of participation and engagement. Fox Sports, Turner Sports, and ESPN all want to engage their audiences, and everyone in the audience has an opinion about what happened. Sports is a no-brainer for audience engagement because that's where the passion lies," said Douglas. "For competition shows, like 'Project Runway,' everybody has an opinion about the characters in the story, so they want to get involved as well. When you gamify it by allowing [viewers] to win prizes and tickets to a premiere, it simply enhances the passion around that particular piece of content."
iHeartRadio uses event-by-event processing to analyze and measure user behavior and experience. It also tracks and measures how releases, features, and changes affect the business. Trending services can be fed into the algorithm used to pick the best songs to be played and to create engaging product features for listeners. The company also benefits from real-time exception handling of abnormal data coming into the system.
"Near real time is an area we are putting a lot of focus on, [for] analytics [and] BI, as well as product features. For our business, this is the area that gives us the biggest bang for our buck," said Lasse Hamre, executive vice president of technology for iHeartRadio, in an interview. "We don't really need sub-millisecond analytics, but having insight into data from one hour, down to the seconds, or 100-millisecond latency, is very important. Currently, a few of our business metrics are powered by near-real-time systems, and iHeartRadio has dashboards that display KPIs."
iHeartRadio also relies heavily on batch analytics and reporting for royalty reporting, churn analysis, cohort analysis, and ad and click analysis. The batch jobs run on a Hadoop cluster, and then the summaries or aggregations are sent to iHeartRadio's data warehouse so the BI and analytics teams have fast access to them. It also uses batch processing as the heavy-lifting mechanism for most of the algorithms that support the iHeartRadio Custom Stations and Live Radio products, as well as for recommendations and personalization. Subsequently, near-real-time systems augment the data to provide the listener with the most up-to-date experience possible. In addition, the data science team relies on batch processing for its analytics and modeling work.
Banks are using real-time analytics to boost financial performance, improve customer experiences, and manage risks. They still run snapshot reports of customers, checking accounts, and loans, but with the rise of mobile banking, bill pay, and social payments, business is moving at a much faster pace.
"Sports, retail, and travel are setting expectations that real-time notification and interaction with a bank is the norm ... so a lot of investment is attacking that particular space," said Jon Nordhausen, vice president of product strategy at financial services technology solutions provider Fiserv, in an interview. "Real-time risk and fraud screening is done today, but you're going to see those capabilities expand in the next few years as we are able to track and capture more information. We [can] build a better customer risk profile as it pertains to their behavior, their payments, their spending habits, and being able to do predictive alerting around potentially fraudulent behaviors."
Banks of all sizes are providing -- or working to provide -- mobile banking and alternative payment options, which offer additional layers of insight into customer behavior.
"Real time is in subseconds. Near real time is less than 10 seconds. When we think about [our] major pillars -- self-service, productivity, customer retention, fraud, and cross-sell -- they're all based on this near-real-time/real-time capability," said Nordhausen. "Within eight to 10 seconds, we have to deliver something relevant to that customer to keep them engaged. In 10 minutes, it's irrelevant, especially in risk and fraud."
Batch processes remain for "close of day" purposes, but those cycles will likely accelerate in the future, Nordhausen said. Meanwhile, banks are using the expanding universe of data to understand how the needs of individual customers will change over the course of a lifetime.
"The more we can predict the journey of individual consumers, the higher the probability we can retain them," said Nordhausen. "Even with data warehouse technology, there are still gaps in terms of being able to retain a lot of information which is needed to create accurate models about current and future behaviors. With technologies like Watson and Hadoop, the cost becomes so cheap, it's down to the creativity to apply new solutions to business problems."
IT departments are using real-time analytics to identify aberrant behavior within systems.
Online customers are notoriously impatient. For the last decade, retailers, banks, and other consumer-focused businesses have been trying to improve online and in-store or in-branch experiences by understanding customer behavior. Now, these businesses want to understand customer behavior in context and respond with a timely, relevant message or offer.
"Web 2.0 is about building user profiles and systems of interaction where the user can check a bill, check an order, or see if something is in stock in real time," said Ryan Betts, CTO of in-memory database provider VoltDB, in an interview. "The next wave of applications are all based upon location, what you just did, and who you are, and they're all pushing information at you right at the moment you require. That's a really different pattern of attraction."
Some of VoltDB's customers have a latency budget of about a fifth of a second. Within about 200 milliseconds, the company has to become aware of an event, make a decision, and push an offer out to a customer's mobile phone before the opportunity window closes.
As Betts said, "the definition of real time is: Can I connect to the customer the microsecond that I've learned something about them, before I lose their attention? Real time means am I still in the scope of the users' intention?"
Millennials are credited with driving customer experience standards because they're digital natives. Their online experience expectations are high, their buying power is increasing as they mature, and they heavily influence pop culture.
Digital branding agency Moncur considers real-time analytics key to its success. It constantly adjusts its publishing and engagement strategies based on the immediacy of results
"Since content across social media tends to have a short half-life, learning what we can in terms of impressions, engagements, and click-though rates for the lifecycle of a post helps empower us to make the adjustments necessary for future content executions," said Moncur principal David Moncur in an interview. "However, there is also a time to consider batch data reporting too. This is usually at the close of a month, when we look at how social platforms for clients and any special campaigns have performed. In this sense, being able to mine data at this level helps us identify anomalies and averages and draw conclusions about how certain audiences behave. If real-time analytics provides us data on a per-post basis, batch analytics allows us to see how we performed overall. With real-time analytics, we can enhance day-to-day tactics, while batch analytics allows us to retool the overall campaign strategy as necessary."
To meet compliance mandates and satisfy commercial and industrial customers, utilities are putting data analytics in place to understand energy consumption patterns in buildings.
Southeastern Ohio Regional Medical Center is an acute healthcare center and community hospital that uses real-time data analytics to reduce the number of hospital-acquired infections and associated costs. One of its upcoming projects involves understanding the real-time status of patient flow in the hospital, including patient registrations and discharges.
"There are multiple items we are attempting to resolve with this. First and foremost, making sure we have the right staff in the right place to move patients through their treatments as efficiently as possible," said Clark Carpenter, infrastructure supervisor at Southeastern Ohio Regional Medical Center. "As an example, we have disbursed registration. If one area is slow and another is backed up, we move a registrar to the backed-up area to get our patients in for treatment, [which increases] patient satisfaction. Patient satisfaction impacts the hospital's bottom line for reimbursement."
Southeastern Ohio Regional Medical Center is also paying attention to the trend data to predict the busy and slow times on daily, monthly, and quarterly cycles.
Information that companies in heavy industries once calculated annually is now calculated in a matter of minutes. Instead of submitting a table of values and list of responses periodically, the underlying information must be accessible on demand and transparent for audit purposes.
"The compliance systems have usually resulted in more spreadsheets with more external links requiring more drive space. This is not sustainable as the complexity and number of environmental regulations increase," said Philip Black, data information management specialist and product marketing strategist at Wood Group Mustang. "In order to meet [the new] requirements, companies are using environmental analytics to turn vast amounts of data into actionable information."
By combining domain-specific knowledge with big data, energy companies can see the relationships between data points that were difficult to pinpoint using spreadsheets.
"In industries such as refining, petrochemical, and manufacturing, where high-frequency measurements and short-term compliance are required, [real-time] environmental analytics are being used to provide predictive guidance as to whether or not operations need to be adjusted to prevent problems. Estimates have always been used, but due to the complexity of the regulatory calculations, they have not been accurate enough to use for predictions," said Black.
Compliance requires faster issue identification and resolution. Batch data historically has been stored in multiple systems, and even if the data was stored in one place it was not necessarily connected to other operational information. Using analytics to review the raw, high-frequency data, hidden issues can be spotted and correlated with external information sources, such as production levels, accounting, or even meteorological data stores.
"Instead of manually sorting through spreadsheets to answer questions such as 'Why do I have more problems in Q3 than Q1?' or 'If production was lower last month, why did we have more compliance issues?' you can simply [run a query]," said Black.
Large companies use real-time and near-real-time analytics to operate more efficiently. And the trend is moving downstream.
Restaurant Management Company is one of the largest Pizza Hut franchisees. It owns 140 locations in Texas, Oklahoma, Louisiana, New Jersey, Colorado, Wyoming, and Montana. The company tracks how long it takes to make a pizza, how long it takes for an order to be processed, how long a pizza sits on a rack before a delivery driver picks it up, how long a delivery driver is on the road, and how many deliveries a driver takes at one time.
"We track everything," said Ken Syvarth, chief operating officer of Restaurant Management Company, in an interview.
The company recently started monitoring pizza ovens, freezers, and HVAC systems, purportedly in real time, to reduce energy costs. It expanded a seven-store SiteSage energy management system pilot to 57 stores. The system monitors events such as when a pizza oven or freezer is turned on or off, and the temperature of the unit, and it controls the HVAC systems. When an operating time or temperature is out of range, an area manager receives an alert about the problem.
"If it's really hot outside, a restaurant manager may crank up the air conditioning to 65 degrees and let it run all day, and at night [she may] forget to turn the air conditioner down, so it's cranking all night long," said Syvarth. Eliminating such energy wasting behavior by providing alerts to managers is "where we see a real savings," he added.
This type of alert is being used in other ways as well. For example, if a walk-in freezer is left off after a person does inventory, a manager is alerted when the temperature reaches a certain threshold in order to prevent the store from losing product.
Large companies use real-time and near-real-time analytics to operate more efficiently. And the trend is moving downstream.
Restaurant Management Company is one of the largest Pizza Hut franchisees. It owns 140 locations in Texas, Oklahoma, Louisiana, New Jersey, Colorado, Wyoming, and Montana. The company tracks how long it takes to make a pizza, how long it takes for an order to be processed, how long a pizza sits on a rack before a delivery driver picks it up, how long a delivery driver is on the road, and how many deliveries a driver takes at one time.
"We track everything," said Ken Syvarth, chief operating officer of Restaurant Management Company, in an interview.
The company recently started monitoring pizza ovens, freezers, and HVAC systems, purportedly in real time, to reduce energy costs. It expanded a seven-store SiteSage energy management system pilot to 57 stores. The system monitors events such as when a pizza oven or freezer is turned on or off, and the temperature of the unit, and it controls the HVAC systems. When an operating time or temperature is out of range, an area manager receives an alert about the problem.
"If it's really hot outside, a restaurant manager may crank up the air conditioning to 65 degrees and let it run all day, and at night [she may] forget to turn the air conditioner down, so it's cranking all night long," said Syvarth. Eliminating such energy wasting behavior by providing alerts to managers is "where we see a real savings," he added.
This type of alert is being used in other ways as well. For example, if a walk-in freezer is left off after a person does inventory, a manager is alerted when the temperature reaches a certain threshold in order to prevent the store from losing product.
-
About the Author(s)
You May Also Like