AI: You Can't Test New Ideas with Old Data - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Data Management // AI/Machine Learning
News
4/20/2018
10:30 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

AI: You Can't Test New Ideas with Old Data

How can you apply machine learning and AI to predict the success of a new idea? Will that old data you have work? Here's a different approach.

Machine learning operations typically rely on quality data to train algorithms. If you are an online retailer, you may look at a customer's past purchases and browsing history and demographics to predict what the "next best offer" should be. The historical data you have collected about customers fuels the algorithm and insights on what your customer is most likely to purchase next.

But while this approach is predictive, it can't let you look at events that haven't happened yet. Just because your customer bought that item every three months doesn't mean they won't find something different to buy in the future or they won't behave differently if the price goes up or down.

(Image: geralt/pixabay)

(Image: geralt/pixabay)

Getting that kind of insight into future behavior is something that Anthony Bruce thinks about every day. His analytics consulting firm, APT, founded in 1999, has worked with giants like Walmart and McDonalds, using another technique that he calls Test and Learn. Here's how it works -- if you operate a sandwich shop, you can't know the effect of discounting a particular sandwich by $2 unless you actually do it.

His approach advocates that kind of testing in different locations and with different demographics and at different times. By testing that discount and gathering the data about its effectiveness at those different geographies and times, your algorithm gets another type of data and can do a better job of learning. After all, it didn't have the data about the discount until after you tested the discount. You can feed all the other historical data into the algorithm, but it won't be able to tell you what effect the discount will have until you actually try it and create the data.

Three years ago, Mastercard saw the value of Test and Learn and acquired APT for $600 million. Mastercard continues to offer what Bruce calls "Machine Experimentation" and "Test and Learn" as it works on a consulting basis with customers. And the company is also including some of those services and intelligence baked into what it offers to all its merchant customers.

For instance, Mastercard is anonymizing and aggregating some data, agreed upon by its merchant customers, to give them benchmark-type insights. So if my restaurant in Boston is down by 4% one week, I might not be so upset about it if I overall Boston restaurants were down 5% for that week. On the other hand, if everyone else is up by 7% and I am flat, that's a real problem. Most of the data, however, remains with each individual customer.

Test and learn everywhere

Like the $2 sandwich discount, the same concept can apply to many other facets of a business, Bruce told InformationWeek in an interview. For instance, a restaurant chain could test a restaurant remodel, or a new type of suburban location, or a new menu item. It could try a new kind of TV advertisement.

"That discipline of rapid innovation through trying new ideas is a necessary complement to modeling," he said. You can't use historical data to model whether that new idea is a good one or not.

Ideas should not be limited by past actions, according to Bruce. For instance, if we are trying to encourage someone to activate their credit card, and we tried three different options, no algorithm will know in advance how a fourth, untested option will work.

These Test and Learn-style techniques most certainly sound familiar to direct marketers who are always testing messaging and have been doing it for years. But these techniques are now being spread to other functional areas and industries, too, Bruce said. That's because digital transformations have created more of a data culture throughout organizations. Now such methods can be applied to staffing, pricing, promotions and really any other activity, according to Bruce.

"We are still in some of the early stages of that, but in pieces of the business that is a familiar concept," he said.

Jessica Davis has spent a career covering the intersection of business and technology at titles including IDG's Infoworld, Ziff Davis Enterprise's eWeek and Channel Insider, and Penton Technology's MSPmentor. She's passionate about the practical use of business intelligence, ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Slideshows
Reflections on Tech in 2019
James M. Connolly, Editorial Director, InformationWeek and Network Computing,  12/9/2019
Slideshows
What Digital Transformation Is (And Isn't)
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/4/2019
Commentary
Watch Out for New Barriers to Faster Software Development
Lisa Morgan, Freelance Writer,  12/3/2019
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
The Cloud Gets Ready for the 20's
This IT Trend Report explores how cloud computing is being shaped for the next phase in its maturation. It will help enterprise IT decision makers and business leaders understand some of the key trends reflected emerging cloud concepts and technologies, and in enterprise cloud usage patterns. Get it today!
Slideshows
Flash Poll