Edge Analytics An Antidote To IoT Data Deluge

6 Machine Learning, AI, Analytics Trends To Watch

Gartner estimates that the Internet of Things (IoT) will include 26 billion devices by 2020. Organizations in virtually every industry are using IoT devices to drive higher levels of efficiency, reduce costs, generate new revenue, and understand customers at more granular levels. However, not all of these organizations are prepared to deal with the deluge of data these IoT devices will bring.

"IoT deployments will generate large quantities of data that will need to be processed and analyzed in real time," said Gartner research director Fabrizio Biscotti in a statement. "Processing large quantities of IoT data in real time will increase as a proportion of workloads of data centers, leaving providers facing new security, capacity, and analytics challenges."

One way of addressing these challenges is to put automated, intelligent analytics at the edge -- near where the data is generated -- to reduce the amount of data and networking communications overhead.

"We definitely don't want to save all data. Trying to decide what to save and what to use is as important as having the facility to capture it," said Nik Rouda, senior analyst at ESG Research.

Rather than sending every bit of data to a centralized location where it can be analyzed, edge analytics places another layer of intelligence and automation where the data resides.

"Having access to all of your data is important, but with access comes responsibility and a need for a strategy about which data needs to be collected at the atomic level, which data needs to be rolled up and aggregated, and which data needs to be used to run your business," said Shawn Rogers, chief research officer at Dell Statistica.

Thinking Differently About Data

Yet, a lot of organizations are saving every bit of data, just in case it might become valuable in the future. From a traditional data center point of view, that's probably not practical over the long term, especially for companies extending out to the IoT.

Some organizations are now creating data lakes to deal with the growing volume of data, whether or not the IoT is part of their strategy. But as always, just because the data can be saved, doesn't necessarily mean it should be saved, unless there's a compelling reason to do so, such as regulatory compliance.

"We're going to see a lot of data collected before people know what to do with it," said Moshe Kranc, CTO of Ness Software Engineering Services. "As the mesh develops and we see all kinds of data being collected and interconnected, there's going to be a need to process it, and it can't possibly be processed in a central location. It has to be processed at the edge."

That's not to say that none of the IoT data will make its way back to a central location, but organizations will have to decide what happens at the edge, at the core, and perhaps in-between. For example, rather than send all sensor data to a central location, an edge device or software solution may send a summary of the data or trigger an automatic alert based a threshold-level status change.

Dell Statistica already has an edge analytics gateway that is capable of communicating with a wide array of sensors. Using its platform users can write a rule-based model that determines how the gateway handles data.

"Our traditional practices have been to collect, harvest, aggregate, and store data, but we're not allowing data to live where it best lives. [Now] we're being more strategic in our approach to the data to decide what data needs to move back to the core and what data can be leveraged for value," said Dell Statistica's Rogers.

Some of the Challenges

Like anything else, the benefits of edge analytics have to be weighed against its costs and risks, which will vary from use-case to use-case.

"Although RFID chips are cheap enough to track boxes in a warehouse, when you're talking millions of boxes, that adds up. And if you think about sensing equipment or a seismograph to study motion in the ground, that can get really expensive if you want to do it over a wide area," said ESG Research's Rouda.

[For more insight on what's coming down the analytics pipeline, see Big Data Goes Mainstream: What Now?]

Security is another issue, because adding another technology layer introduces another potential point of vulnerability. In addition, the data being collected may raise privacy concerns, by itself or when it's combined with other kinds of data. In other words, data governance may need some adjustment.

To be of value, edge analytics should be part of a holistic data strategy. Otherwise, companies run the risk of creating an information silo, which would be a huge step backward from both business and IT perspectives.

"Whenever you acknowledge that data should live where it needs to, you run the risk of creating a siloed environment if you don't have a strategy that includes smart data integration and management, an infrastructure that can support it, and end-to-end security that covers your entire ecosystem," said Rogers.

Bottom Line

The perpetual questions of what data can be collected, what data should be collected, and how long the data should be retained still apply. The difference is the physical point at which the data should be analyzed and acted upon, which depends on the use-case and on what an organization is trying to achieve.

"I think a lot of data can be understood locally. Edge analytics means you don't need to bring everything to the data center and save it forever. Maybe we can interpret it, respond to it, record it, or summarize it, and then share it with other locations," said Rouda.

While we'll likely hear more about edge analytics in the near future, not everyone will agree that it's necessary. As processing power and storage costs continue to fall, and network bandwidth continues to increase, organizations may decide that they can keep every bit of their data regardless. Whether that actually makes sense is a question that organizations can only answer for themselves.