Real-Time Prediction, Big Data Scale: Impossible Dream? - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Software // Information Management
09:06 AM
Doug Henschen
Doug Henschen
Connect Directly

Real-Time Prediction, Big Data Scale: Impossible Dream?

WibiData challenges IBM, Oracle, and SAS by combining vast customer history as well as real-time behavioral data to generate personalized recommendations.

Marketers have been talking about one-to-one marketing for decades, and the challenge is that much harder now that we can tap into big data.

Pioneering startup WibiData says it's bringing big retailers and financial institutions the best of both worlds, combining vast data stores from Hadoop and HBase with the very latest customer behaviors captured on websites and mobile apps. Best of all, application developers don't have to become data scientists to take advantage of the real-time recommendations.

"Front-end Web developers shouldn't have to think about how a predictive model is constructed, calculated, or applied," says Christophe Bisciglia, CEO and co-founder of WibiData and a co-founder of Cloudera. "They should just be able to get recommendations, take the results, and apply them however they want."

WibiData's differentiator is bringing both big data and real-time data into individualized recommendations for specific customers (not customer segments) as they interact with websites, mobile apps, in-store kiosks, or any other channel. The product behind that promise is WibiEnterprise 3.0, which was released late last month.

[Want more on next-generation demands? Read 5 Big Wishes For Big Data Deployments.]

WibiData's competition includes the likes of Oracle RTD (Real-Time Decisions); web-based services like Certona, iGoDigital, and Strands Recommender; and home-grown systems. But these options invariably can't handle big data -- such as log files or customer web or mobile clickstreams -- or they can't individualize the recommendation based on the latest browsing experience, according to Bisciglia.

RTD and online recommendation engines are "great for helping you understand that customers who bought X also bought Y," Bisciglia says, "but that doesn't capture the intent that the customer is expressing in that current session. It won't tell you that somebody is shopping for a gift, not buying what they normally buy. And it won't tell you that the customer just purchased a TV, so stop showing them other TVs and start showing them HDMI cables and speaker systems."

WibiData counts big retailers and global financial services among its customers, but these customers won't let their names be used, according to Bisciglia. Among the handful of WibiData customers talking about their deployments publicly is Opower, a company that analyses utility smart-meter data in real time to help customers understand and reduce their power consumption.

"Trying to solve big data challenges by combining [Hadoop and HBase] with traditional application development processes mires even the most talented developers and data scientists in tedious and laborious modeling and development cycles,"said Rick McPhee, Opower's SVP of Engineering, in a recent statement. "After facing some of these problems first hand, we’ve worked with WibiData to simplify and speed our big data application development processes."

WibiData has been working on the problem since its founding in 2010, but it has morphed along the way from providing a monolithic, proprietary product to offering enterprise support for modularized, open-source software. The Apache-licensed software is the Kiji Project framework, which works on top of Hadoop and HBase and includes two main components. Kiji Express is an analytical language in which you can build models or integrate with models developed in other languages, such as SAS or SPSS, using PMML (predictive model markup language). The models are executed in the Kiji Scoring Server, which has a REST interface that lets developers plug the models into web and mobile applications without having to know a thing about predictive modeling.

There are certainly others on the list of vendors working on the combination of big data and real-time, predictive analytics -- IBM, Pivotal, SAS, Teradata -- but Bisciglia claims cutting-edge data scientists are turning to a new generation of vendors to make it happen.

"This new breed of data scientists tends to use tools like R and Python and Scala," he observes. "We're not seeing them use the traditional tools and we're not talking about doing SQL querying on top of Hadoop."

Associating SQL with BI, Bisciglia says WibiData Enterprise customers can use the product's Hive adapter to query, say, how many times a customer clicked on an offer. But to build a predictive model that will present that customer with the right offer based on their history and their current session behavior? For that, WibiData's prescription is the Scala-based Kiji language to develop the model, its schema-management and model lifecycle-management tools, and its REST interfaces to capture real-time information and to plug the model into web or mobile applications. 

With giants like IBM, Oracle, SAS and others pursuing real-time e-commerce at big-data scale, it might be hard to think about a little startup with a funny name like WibiData as much of a threat. But Bisciglia knows all too well that that's what they said about Hadoop and Cloudera three or four years ago.

Doug Henschen is executive editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data, and analytics. He previously served as editor-in-chief of Intelligent Enterprise, editor-in-chief of Transform Magazine, and executive editor at DM News.

IT groups need data analytics software that's visual and accessible. Vendors are getting the message. Also in the State Of Analytics issue of InformationWeek: SAP CEO envisions a younger, greener, cloudier company (free registration required).


We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
D. Henschen
D. Henschen,
User Rank: Author
12/17/2013 | 4:29:01 PM
Re: There are plenty of other options...

Thanks for the comment. "Micro-contextual segmentation consumer dossier at the touch point within the existing CRM workflow..." is quite a mouthful, but what is it? Is that synonymous with one-to-one-predictive offers? Wibi's claim is that's it's getting away from segments and predicting the right offer for individual customers. The latest interaction info is even more relevant, because if the offer is for a TV, but they just bought one, what good it the prediction?
User Rank: Author
12/17/2013 | 12:50:40 PM
Real-time shopping intent
I always smile when i log into Amazon and it suggests "a little something for you," except it is based on gift purchases I have made for boys under the age of 10. So WibiData promises to figure out I am shopping for a gift, not for myself? What are the clues?
D. Henschen
D. Henschen,
User Rank: Author
12/17/2013 | 11:05:22 AM
There are plenty of other options...
Beyond all the well-known vendors mentioned above, plenty of integrators and other startups are working on combining big-data analytsis and real-time prediction. WibiData is working toward productizing this capability and making it less of a one-off, reainvent the wheel proposition. It still requires plenty of development work on the app side and data science on the analytics side, but at least you're not starting entirely from scratch. 
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Becoming a Self-Taught Cybersecurity Pro
Jessica Davis, Senior Editor, Enterprise Apps,  6/9/2021
Ancestry's DevOps Strategy to Control Its CI/CD Pipeline
Joao-Pierre S. Ruth, Senior Writer,  6/4/2021
IT Leadership: 10 Ways to Unleash Enterprise Innovation
Lisa Morgan, Freelance Writer,  6/8/2021
White Papers
Register for InformationWeek Newsletters
Current Issue
Planning Your Digital Transformation Roadmap
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Flash Poll