12 Barriers To Real-Time Analytics
As organizations look to stay competitive by expanding their use of real-time analytics, implementation becomes a challenge. Finding options to effectively serve your company over the long term is often more difficult than it appears. We've identified 12 common obstacles you'll want to avoid as your company pursues real-time analytics.
What does "real time" analytics mean to you and your business?
IT departments have been bombarded with requests for real-time analytics capabilities for years, although not everyone who asks for it actually needs it -- or even knows what it means. Is the high cost of moving to real-time analytics justified? Although the prices of memory, storage, and bandwidth all continue to fall, there are technology integration issues, process issues, and cultural issues to be considered.
Adding to the confusion, there is no standardized definition of "real time." Depending on whom you ask, "real-time" can be measured in anything from sub-seconds to a span of more than 24 hours.
Technology innovations such as in-memory computing, Hadoop, and Spark are all designed to address the insatiable need for speed. Yet, a number of persistent issues keep companies from moving as fast as business leaders might desire.
[What's hot in data science? Read 10 Programming Languages and Tools Data Scientists Use Now.]
"IT organizations often do not have the platforms in place to collect, manage, and respond to real-time data. This means any attempt at real-time [action] will be flawed and even dangerous," said Graham Clark, head of digital services for global IT solutions service provider NIIT Technologies, in an interview. "Additionally, creating and using platforms for real-time analytics requires a real-time business mentality, which needs to permeate throughout the organization."
Companies should consider the use-case instead of focusing on real-time analytics as the goal for all projects, said Jeff Veis, VP of big data platform marketing for HPE Software at Hewlett Packard Enterprise, in an interview. "After all, it largely depends on how quickly you need an answer, if the business can wait for that insight -- and at what cost."
Increasingly, businesses are considering the cost of not implementing real-time analytics. These days, response time is considered a competitive weapon, and some observers believe this trend will become even more pronounced with the proliferation of Internet of Things (IoT) devices. However, accelerating business processes introduces risks that must also be weighed against the potential benefits.
"People might go to extremes and demand all available data to be collected and used on a real-time basis, [but] not all data is equally important," said Vivian Zhang, founder and CTO of the NYC Data Science Academy, in an interview. "Some data is critical, some data isn't. When you put too much noise or irrelevant information into the data pipeline, it might not help and it can result in a false impression of what is happening inside the business."
In short, don't underestimate what it takes to do real-time analytics right, because there's a lot that can go wrong along the way. Following are a few of the most common barriers to success. Once you've reviewed these, tell us about your own experiences.
What does "real-time" analytics mean to your organization? How close are you to achieving it? What pitfalls have you encountered? Tell us all about it in the comments section below.
Business executives are attracted to vendors who pitch the promise of real-time analytics software without all the IT headaches and policies. It seems simple enough. Regardless of who is purchasing and implementing real-time analytics capabilities, it is wise to understand what users consider real-time, because their definitions can differ by one or more orders of magnitude.
"Real-time means instantaneous, but we believe real time is as [fast] as possible. Most importantly, [it's] as real time as the customer needs," said Graham Clark of NIIT. "A financial services trading company might be obsessed with microseconds and nanoseconds. A customer looking for a cheap airline ticket might have a two-month window. Start with the customer need and evolve accordingly."
Centralized IT often serves as a bottleneck to progress. At the same time, the phenomenon of Shadow IT -- the ability for business units to make some software purchasing decisions on their own, without involving IT -- is causing fragmentation. Resulting problems can be difficult, costly, time-consuming, and highly political to rectify.
"There's a question of subtle balance that may be hard to achieve," said Andrew Brust, senior director of Market Strategy and Intelligence at Big Data analytics platform vendor Datameer, in an interview. "The line of business needs autonomy so it can get its work done, and that autonomy can benefit IT by removing deliverables from its backlog.
"But IT must be in the loop, able to understand and monitor what the line of business is doing, and confirm the efficacy of its data and analysis. Too often, companies find themselves without this balance, where the line of business is working in a somewhat rogue fashion, or where IT is acting as a huge bottleneck to progress."
Moving data around without proper governance controls in place can expose your organization to unwanted risks. Shadow IT is often to blame.
"As with any data governance strategy, data must be tested and validated before it is folded into the overall BI practice," said Ulrik Pedersen, CTO of BI and analytics software vendor TARGIT, in an interview.
"With real-time data, this is hard to do because there is no time to clean the data before it makes its way to the users. That's why it's critical to write queries that can handle data issues such as missing values, missing records, other data entry errors, and calculations that are intelligent enough to couple with outliers."
You think you need real-time data. But how long will it really take your organization to act on the insights provided? Sure, some real-time systems include automated decision-making. But, in a decision-support scenario, a human's ability to react to real-time information suggests true real-time capabilities aren't so practical.
"In my opinion, it's a lot like "The Emperor's New Clothes." Most of it is about appearing to be in the know, rather than responding to a real need," said Marcelo Burzstein, president of digital experience firm 76Design. "In fact, I find that the more people understand about real-time analytics and predictive methods, such as linear regression and other machine learning algorithms, the more they realize real-time analytics may not be the best model for their use-cases."
Technology innovation has always moved faster than business culture. Existing infrastructure and the need to move quickly can sometimes be at odds, either slowing the flow of data or creating headaches that only emerge further down the road.
"IT has made huge investments in traditional data warehouse and BI systems, which cannot always support real-time use-cases," said Ramesh Hariharan, head of innovation and technology at analytics software vendor LatentView Analytics, in an interview. "Another reason is the process used to deliver these capabilities. Business is more focused on agility, perhaps at the expense of scalability and other enterprise-level considerations."
Is real-time analytics merely another IT project, or is it a crucial enterprise capability? Viewing it as yet another project to be tackled can be damaging over the long term. Don't start out with a huge costly project that will take two or three years to implement. Requirements will change along the way.
"Operating a real-time analytics-driven business model which delivers real-time predictions and real-time prescription ... is a brand-new way of thinking," said Graham Clark of NIIT Technologies. "There is a widely held belief that real-time analytics outcomes are achieved through a project, whereas our experience is that successful real-time analytics is an initial project followed by an ongoing and perpetual optimization process."
It's the C-suite's job to understand which specific problems are being solved, which specific business opportunities are being pursued, and the quantifiable results of those undertakings. Unfortunately, C-level executives often have unrealistic expectations about what it takes to deliver real-time analytics.
"[The C-suite] should understand that real-time analytics is not an overnight success, but built on a foundation of enterprise-wide investments in good data infrastructure," said Ramesh Hariharan of LatentView Analytics, in an interview. "Real-time analytics is built on the foundation of real-time data collection and data movement (equivalent to ETL). Availability of technology is usually not the barrier. The way the technologies are viewed and used is the barrier."
There are a lot of barriers to real-time information delivery. In-memory vendors are addressing response times. Improved APIs are helping to tackle integration issues. Hadoop and the cloud are tackling capacity issues. Are these the best options for real-time analytics use-cases?
"Perhaps the biggest false expectations that business users have about real-time analytics [are] about scale and the capability of a real-time system to be open. Because real-time analytics must be executed at precise intervals, introducing new data feeds, more data, or new analytics to the system may disrupt that timing," said Jeff Veis of Hewlett Packard Enterprise. "Companies should not expect that real-time analytics has unlimited scale, [or that it] can support unlimited users or an unlimited quantity of data."
Any technology project not tied back to some sort of ROI (cost/benefit analysis) and metrics is destined to fail. When it comes to real-time analytics, IT tends to look at the cost and the business tends to look at the opportunities. What's often overlooked is an opportunity cost, which is the cost of failing to act.
"The C-suite should understand how much money they're losing the longer they wait to respond to business phenomena and put new policies into action," said Andrew Brust of Datameer. "Do some gaming of the scenarios and see if acting sooner would significantly and positively impact the bottom line. If there are risks to responding too quickly, those should be factored in as well. This will help set expectations and requirements, and provide a realistic sense of project budgets and overrun tolerances."
Real-time analytics helps enable new business models. Companies, and entire industries, are being disrupted by innovators who are using capabilities, including real-time analytics, to change the rules of well-entrenched industries.
"Many C-suites missed the Uber revolution, which is why they did not have the means to conceptualize real-time design support needs," said Rado Kotorov, CIO of enterprise BI software company Information Builders. "They have to think about which kinds of real-time decisions can bring more business."
-
About the Author(s)
You May Also Like