Real-Time Prediction, Big Data Scale: Impossible Dream?

WibiData challenges IBM, Oracle, and SAS by combining vast customer history as well as real-time behavioral data to generate personalized recommendations.

Doug Henschen, Executive Editor, Enterprise Apps

December 17, 2013

4 Min Read

Marketers have been talking about one-to-one marketing for decades, and the challenge is that much harder now that we can tap into big data.

Pioneering startup WibiData says it's bringing big retailers and financial institutions the best of both worlds, combining vast data stores from Hadoop and HBase with the very latest customer behaviors captured on websites and mobile apps. Best of all, application developers don't have to become data scientists to take advantage of the real-time recommendations.

"Front-end Web developers shouldn't have to think about how a predictive model is constructed, calculated, or applied," says Christophe Bisciglia, CEO and co-founder of WibiData and a co-founder of Cloudera. "They should just be able to get recommendations, take the results, and apply them however they want."

WibiData's differentiator is bringing both big data and real-time data into individualized recommendations for specific customers (not customer segments) as they interact with websites, mobile apps, in-store kiosks, or any other channel. The product behind that promise is WibiEnterprise 3.0, which was released late last month.

[Want more on next-generation demands? Read 5 Big Wishes For Big Data Deployments.]

WibiData's competition includes the likes of Oracle RTD (Real-Time Decisions); web-based services like Certona, iGoDigital, and Strands Recommender; and home-grown systems. But these options invariably can't handle big data -- such as log files or customer web or mobile clickstreams -- or they can't individualize the recommendation based on the latest browsing experience, according to Bisciglia.

RTD and online recommendation engines are "great for helping you understand that customers who bought X also bought Y," Bisciglia says, "but that doesn't capture the intent that the customer is expressing in that current session. It won't tell you that somebody is shopping for a gift, not buying what they normally buy. And it won't tell you that the customer just purchased a TV, so stop showing them other TVs and start showing them HDMI cables and speaker systems."

WibiData counts big retailers and global financial services among its customers, but these customers won't let their names be used, according to Bisciglia. Among the handful of WibiData customers talking about their deployments publicly is Opower, a company that analyses utility smart-meter data in real time to help customers understand and reduce their power consumption.

"Trying to solve big data challenges by combining [Hadoop and HBase] with traditional application development processes mires even the most talented developers and data scientists in tedious and laborious modeling and development cycles,"said Rick McPhee, Opower's SVP of Engineering, in a recent statement. "After facing some of these problems first hand, we’ve worked with WibiData to simplify and speed our big data application development processes."

WibiData has been working on the problem since its founding in 2010, but it has morphed along the way from providing a monolithic, proprietary product to offering enterprise support for modularized, open-source software. The Apache-licensed software is the Kiji Project framework, which works on top of Hadoop and HBase and includes two main components. Kiji Express is an analytical language in which you can build models or integrate with models developed in other languages, such as SAS or SPSS, using PMML (predictive model markup language). The models are executed in the Kiji Scoring Server, which has a REST interface that lets developers plug the models into web and mobile applications without having to know a thing about predictive modeling.

There are certainly others on the list of vendors working on the combination of big data and real-time, predictive analytics -- IBM, Pivotal, SAS, Teradata -- but Bisciglia claims cutting-edge data scientists are turning to a new generation of vendors to make it happen.

"This new breed of data scientists tends to use tools like R and Python and Scala," he observes. "We're not seeing them use the traditional tools and we're not talking about doing SQL querying on top of Hadoop."

Associating SQL with BI, Bisciglia says WibiData Enterprise customers can use the product's Hive adapter to query, say, how many times a customer clicked on an offer. But to build a predictive model that will present that customer with the right offer based on their history and their current session behavior? For that, WibiData's prescription is the Scala-based Kiji language to develop the model, its schema-management and model lifecycle-management tools, and its REST interfaces to capture real-time information and to plug the model into web or mobile applications. 

With giants like IBM, Oracle, SAS and others pursuing real-time e-commerce at big-data scale, it might be hard to think about a little startup with a funny name like WibiData as much of a threat. But Bisciglia knows all too well that that's what they said about Hadoop and Cloudera three or four years ago.

Doug Henschen is executive editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data, and analytics. He previously served as editor-in-chief of Intelligent Enterprise, editor-in-chief of Transform Magazine, and executive editor at DM News.

IT groups need data analytics software that's visual and accessible. Vendors are getting the message. Also in the State Of Analytics issue of InformationWeek: SAP CEO envisions a younger, greener, cloudier company (free registration required).


About the Author(s)

Doug Henschen

Executive Editor, Enterprise Apps

Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of Transform Magazine, and Executive Editor at DM News. He has covered IT and data-driven marketing for more than 15 years.

Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.

You May Also Like

More Insights