How to Integrate Edge Computing With Your Core Systems

Between now and 2025, the edge computing market is expected to grow by a compound annual growth rate (CAGR) of 19.9%. It’s small wonder. Companies are deploying the Internet of Things at the edges of their enterprises, in homes, and in the field. These devices can send, receive and process data, with a next wave of deployment that is likely to focus on how to derive and process data for business value out of all of this IoT.

In some cases, the IoT will be able to receive, process and even store data on its own. In other cases, it will be required to cooperatively share data with other core enterprise systems to support technologies like artificial intelligence and analytics.

[For more about edge computing see how the cloud plays a key role: Exploring Edge Computing as a Complement to the Cloud]

Regardless of the IT architecture a company develops for its edge/IoT and its centralized core computing, being able to integrate all of these data sources for business results will be an important focus.

The edge-core integration challenge

For IT, there are five key challenges for integrating edge and core computing.

1) Edge (and especially IoT) data overload

IDC is projecting that there will be 41.6 billion connected IoT devices generating 79.4 zettabytes (ZB) of data by 2025. Not all of this data will be useful to companies, and companies can ill afford to manage it all.

“Businesses risk being overwhelmed by IoT data that has little value to the enterprise,” said Manish Jethwa, CTO at Yotta, an infrastructure asset management company, in a published interview. “Without a clear understanding of what data resides where, the Internet of Things (IoT) could be overrun with masses of data that has [sic] no business value.”

2) Limited time to integrate

Speed to market for applications is what the business wants. However, ask any CIO, and he or she is likely to tell you that system integration, needed for most applications, is one of the most difficult, risky and time-consuming projects that IT is tasked with. Edge computing adds to the complexity, since many edge devices come with their own proprietary operating systems, and don't necessarily interface with other IoT devices or with other central IT systems.

3) Legacy systems

Two years ago Chris O’Malley, the CEO of Compuware, a mainframe solutions provider, wrote that  “Fifty-seven percent of mainframe users currently run more than half of their business-critical applications on the platform, with this number expected to increase to 64% by next year.”

My own work with enterprises confirms this. One reason legacy systems are called “legacy” is  because they’ve stood the test of time with incredible reliability and performance. In fact, one CIO of a large hospitality company told me that the mainframe that processes all of his hotel reservations “hasn't gone down” for 30 years.

Reliable legacy systems are invaluable assets in a 24/7 world. But when it comes to integrating legacy systems with edge and IoT, it can be tough going. Legacy systems weren't originally designed for IoT. Many legacy systems also contain thousands of lines of custom code, some of it “black box” because no one currently on staff understands it and it lacks documentation. All of this complicates edge integration

4) Bandwidth

IoT data is continuously being generated at the edge, but if you want it to go elsewhere do you have the bandwidth to support these mega data streams? For IT, this is both a financial and a technical concern.

5) Security

Recent surveys show  that nearly half of US companies using IoT have experienced security breaches. One of the issues is the people using IoT devices, or the non-IT personnel in charge of securing them at the edge. A second challenge is the diversity of edge and IoT infrastructure. The edge can be a network node, a sensor, a gateway, hardware, or application software. There are many moving pieces and many different vendors providing them. The end results are more vulnerabilities and heightened security risk.

Defining an effective integration strategy for the edge

The desire in any integration effort between central computing and the edge is to avoid laborious custom coding at all costs. It is is why the first question that system integrators ask themselves is whether an extensive API library exists for IoT/end-point and core systems. Secondary choices include finding an ETL (extract, transform, load) tool that can help automate edge-core system integration, or perhaps decoupling edge and central computing so the two can operate independently of each other, and then using a cloud intermediary to pass data between them. The options:

APIs. There are core computing systems that come with hundreds of application programming interfaces (APIs) that are a great help in integrating disparate systems with edge applications. The “catch” is that not all of the edge’s proprietary systems and protocols are covered.

Recommendation: If at all possible, use a standard API to make your edge connections. If this fails, proceed to one of the following techniques.

ETL. Extract, transform and load software is able to take data from one system (i.e., an appliance at the edge), and transform the data into a format that is acceptable to a core central system. The transformed data is then loaded into the target central system(s).

The ETL can operate autonomously with the business rules for data transformation that you provide. Extraneous edge data that your central systems don't require can be excluded. With ETL, based upon the business rules you input, data can also be cleaned and normalized.

Recommendation: Use ETL when APIs are non-existent or inadequate. One advantage of an ETL is that it can do so much more with the business rules for data transformation that you give it.

Offload processing to the edge. If you want to conserve bandwidth and also distribute processing so that more of it occurs directly on the edge, there are technologies that do this.

“We can do this by deploying smart edge nodes at the edge for localized processing,” said Kurt Michel, senior marketing VP at Veea, one company that provides this.

In a configuration like this, your edge processes incoming edge data. Subsequently, you can refine this data in the cloud, eliminating the data that you don’t need, and then send it to core systems.

In essence, you are decoupling your processing so the edge can run on its own. This reduces bandwidth consumption. At the same time, it can use other intermediate assets such as a public or private cloud to refine the data that is collected at the edge before the data is loaded into core systems.

Recommendation: Consider localized processing at the edge if you want to conserve bandwidth and offload processing from your core systems.

Custom coding. When all else fails, there is still a place for custom coding. Not every integration situation between the edge and core computing can be solved by commercial software.

The good news is that custom coding is fast becoming the exception and not the rule for edge and core computing integration.