How 2020 Impacted 2021's Predictive Modeling

Before the pandemic hit, digital disruption was the major force shaping the direction of business models and entire industries. As 2020 demonstrated, digital disruption seems relatively tame now compared to COVID-19's impacts on business. The sudden and dramatic shifts in everyday realities negatively impacted predictive model accuracy because they were so inconsistent with historical data.

"One of the really big things that people grappled with is the fact that they took for granted that the models were built properly," said Scott Zoldi, chief analytics officer at decisioning platform provider FICO. "Obviously we were in a time of huge stress so as people were trying to understand how to pivot their business, instead of asking, 'How can I leverage the asset I have?' they basically said, 'Let's just throw the model out and build a new model’, which comes with a whole set of other issues because we essentially then have models built on nonstationary data."

Adapting to the New Normal

Companies have traditionally had years of data that could be used for predictive purposes. However, when the entire world changes so radically in such a short time -- supply and demand, supply chain disruption, shuddered businesses, stay at home mandates -- it's time to get creative.

"A traditional approach would be to look at sales and understand trends. Now sales is not a good predictor so you have to look for something else," said Dan Simion, VP of AI and analytics at global consulting firm Capgemini North America.

Dan Simion, Capgemini

For example, one of Capgemini's airline clients is using future bookings to predict business travel instead of sales because the demand for business travel evaporated in 2020.

While companies understood they needed the ability to adapt to change quickly, they also understood they needed to minimize risks by using data to make decisions.

"Trying to predict using traditional statistical techniques gets harder and harder because you need a lot of data points and observations," said Simion. "Before you would have one observation each week or each day and you could go down to every hour."

The same thing holds true for other dimensions that can be decomposed into smaller pieces -- zip codes instead of countries or regions, for example.

"That is increasing the degrees of freedom, the number of observations within the same time frame," said Simion.

Contingency Planning Is "In"

At a business strategy level, organizational leaders were warned that they needed to do contingency planning at an entirely different level than they had before. Instead of having a plan A and a plan B, global consulting organizations were advising clients to have several contingency plans covering different scenarios such as lock downs and supply chain disruptions. However, the same type of thinking didn't trickle down to the data team in many organizations.

"We're seeing a pickup in demand, especially lately," said Simion. "The question used to be, 'What is the contingency plan?' and now it's 'What are my options for shipping route if I can't ship through traditional routes? Where should I place my containers to account for that?"

Why FICO's Predictive Models Weathered the Pandemic Better Than Most

FICO had fewer challenges with its predictive models in 2020 than most other organizations. Then again, customers rely on its models to make important business decisions such as whether to issue credit and at what level.

Scott Zoldi, FICO

"Prior to COVID, we were always criticized. Why do your models take so long to build when this Fintech over here can do it in the cloud [a lot faster]?" said Zoldi. "We would say to the client, 'You and I both depend on this model and therefore we have to understand it carefully and we have to build it carefully."

Part of FICO's secret sauce is a four-prong methodology that includes:

Zoldi also underscored the importance of a governance model or model development governance model.

"If you don't have a process written down and codified to establish that from this point forward, we're only going to use these technologies, have these kinds of people review the model, these are the standards for what it means to build a robust and responsible model, and out of that would come things the organization would want to monitor to make sure the model is performing properly," Zoldi said.

In a forthcoming report sponsored by FICO, 90% of the CIOs, chief data officers and chief AI officers surveyed said they have to make fundamental changes and investment in how they monitor their models.

"I think if 90% of analytics leaders in these different firms say we have a huge amount of work to do in monitoring I think that's probably one of the big things to look at in 2021," said Zoldi. "The other thing to focus on in 2021 is that if models are built properly and carefully, you don't lose their predictiveness but their interpretation changes a little bit meaning you might use a different score threshold than you did before."

Related Content:

Making Predictive Analytics Work in an Uncertain World

How IT Can Get Predictive Analytics Right

IoT and Predictive Analytics: What We’re Driving Toward

Why Everyone's Data and Analytics Strategy Just Blew Up