Parallel Computing Skills Boost IT Arsenal

Parallel computer processing is used for big data, analytics, artificial intelligence and even processors on desktops and laptops.

It’s not really new. The first parallel processing projects began in the 1960s, and were so-named because they ran multiple streams of calculations and data processing tasks through multiple central processing units (CPUs) that were working simultaneously.

Large banks like Wells Fargo and JP Morgan Chase use parallel processing for financial risk management for their portfolios; and Oak Ridge National Laboratory is using it to assess the likelihood of mental illness in children. Along with universities, research centers and large enterprises that depend upon high-speed parallel processing of large amounts of data for their businesses (e.g., the pharmaceutical, oil and gas, retail and life sciences industries), these corporate users have the financial resources and internal talent to run parallel processing applications on their own. However, there are also thousands of other companies that now want AI, big data processing and analytics, but they don’t have the same compelling use cases or the financial flexibility to develop internal talent for parallel processing environments.

What do they do?

Outsourcing: The Logical Answer

The most common-sense answer for most organizations is outsourcing big data and parallel processing to cloud service providers that have the muscle and the talent to run it. These cloud providers might be “turnkey” in the sense that they provide both the parallel processing and the application; or they could be co-location data centers where IT simply hosts its applications.  In both of these cases, there is a cogent argument that IT doesn’t really need on-staff knowledge of parallel processing, because all that is needed is access to a cloud service that can do it for you.

Is Outsourcing All There Is to it?

For most companies, outsourcing parallel processing work is the answer for using AI, analytics, or any other application that requires the parallel processing of vast stores of data.

However, there are still companies that are caught in the middle. They aren’t so large or wealthy that they can afford a full staff of parallel processing system experts, but the importance of their parallel processing use cases makes it imperative for IT to at least have some on-staff expertise in parallel processing platforms and the work that they facilitate. In these cases, IT needs internal expertise.

The four areas that IT most likely would address are:

Hiring Parallel Processing Talent

More than likely, parallel processing skills will not be found in the organization. These skills aren’t easy to acquire through normal recruiting channels, either.

However, universities are teaching parallel processing programming and system skills.

One approach IT could take would be to partner with universities that have programs in parallel computer processing. Companies could work with professors and offer internships to promising students. Those students who excel in their internships could be hired by the company into permanent positions for parallel processing development and support.

The skills these hires would bring to the table would include knowing how to analyze and fine-tune parallel processing and high-performance software to solve complex problems, execute algorithms and process large amounts of data quickly. The recruits would also know how to optimize multithread streams of data and AI for best performance.

Assigning Business Analysts to the Parallel Processing and AI Environment

Over time, the analytics produced by parallel processing will drift away from the original intent of the business. This might be caused by data becoming less relevant or by the business use case objectives changing.

A parallel processing operation run by a cloud services provider won’t be sensitive to this. It is up to an IT business analyst to work with users by checking and re-checking results from processing for accuracy, and then revising data feeds and/or algorithms as needed.

Using Business Analysts and Database Personnel to Narrow Data Funnels

A couple of years ago, a parallel processing project was being run on a particular Covid-19 molecule in Europe. The goal was to isolate the elements of this molecule to come up with a vaccine recipe.

At the onset of the project, documents and artifacts from every corner of the world were fed into a supercomputer for evaluation. Researchers soon concluded that it was necessary to reduce the amount of raw data being fed into the computer to save time.

The researchers chose to narrow the funnel of the data being admitted into the computer for processing to only those documents and artifacts that named the molecule they were studying. The technical processor of the data didn’t know how to do this -- but the subject matter experts on the user side did.

This is the same challenge that companies outsourcing parallel processing experience. They must use subject matter experts, business analysts and database analysts to determine how to narrow the funnel of incoming raw data, and then direct their cloud processor.

Monitoring Resource Consumption

IT storage, networking and system professionals should regularly assess parallel processing costs and performance in the cloud. If costs are exceeding budget, is there a good reason for it? If there isn’t a good reason, what actions do you take to reduce costs?  To answer and act on these issues, systems, storage and networking professionals must acquire a fundamental knowledge of parallel processing and how it works.

Summary Remarks

Most companies will choose to outsource their parallel processing, AI and analytics work to cloud suppliers, but this doesn’t mean that IT won’t require some on-staff skills and expertise in the parallel processing realm.

The staff members in IT who are cross-trained for parallel processing don’t have to be data scientists, but they should have sufficient knowledge of the parallel processing world and the analytics and AI that run on it. Skilling up staff for parallel processing is an open item that CIOs must address.

What to Read Next:

How New Chip Innovations Will Drive IT

What You Need to Know About Neuromorphic Computing

IT Skills: Top 10 Programming Languages for 2023