AI: You Can't Test New Ideas with Old Data

Machine learning operations typically rely on quality data to train algorithms. If you are an online retailer, you may look at a customer's past purchases and browsing history and demographics to predict what the "next best offer" should be. The historical data you have collected about customers fuels the algorithm and insights on what your customer is most likely to purchase next.

But while this approach is predictive, it can't let you look at events that haven't happened yet. Just because your customer bought that item every three months doesn't mean they won't find something different to buy in the future or they won't behave differently if the price goes up or down.

Getting that kind of insight into future behavior is something that Anthony Bruce thinks about every day. His analytics consulting firm, APT, founded in 1999, has worked with giants like Walmart and McDonalds, using another technique that he calls Test and Learn. Here's how it works -- if you operate a sandwich shop, you can't know the effect of discounting a particular sandwich by $2 unless you actually do it.

His approach advocates that kind of testing in different locations and with different demographics and at different times. By testing that discount and gathering the data about its effectiveness at those different geographies and times, your algorithm gets another type of data and can do a better job of learning. After all, it didn't have the data about the discount until after you tested the discount. You can feed all the other historical data into the algorithm, but it won't be able to tell you what effect the discount will have until you actually try it and create the data.

Three years ago, Mastercard saw the value of Test and Learn and acquired APT for $600 million. Mastercard continues to offer what Bruce calls "Machine Experimentation" and "Test and Learn" as it works on a consulting basis with customers. And the company is also including some of those services and intelligence baked into what it offers to all its merchant customers.

For instance, Mastercard is anonymizing and aggregating some data, agreed upon by its merchant customers, to give them benchmark-type insights. So if my restaurant in Boston is down by 4% one week, I might not be so upset about it if I overall Boston restaurants were down 5% for that week. On the other hand, if everyone else is up by 7% and I am flat, that's a real problem. Most of the data, however, remains with each individual customer.

Test and learn everywhere

Like the $2 sandwich discount, the same concept can apply to many other facets of a business, Bruce told InformationWeek in an interview. For instance, a restaurant chain could test a restaurant remodel, or a new type of suburban location, or a new menu item. It could try a new kind of TV advertisement.

"That discipline of rapid innovation through trying new ideas is a necessary complement to modeling," he said. You can't use historical data to model whether that new idea is a good one or not.

Ideas should not be limited by past actions, according to Bruce. For instance, if we are trying to encourage someone to activate their credit card, and we tried three different options, no algorithm will know in advance how a fourth, untested option will work.

These Test and Learn-style techniques most certainly sound familiar to direct marketers who are always testing messaging and have been doing it for years. But these techniques are now being spread to other functional areas and industries, too, Bruce said. That's because digital transformations have created more of a data culture throughout organizations. Now such methods can be applied to staffing, pricing, promotions and really any other activity, according to Bruce.

"We are still in some of the early stages of that, but in pieces of the business that is a familiar concept," he said.