Software // Information Management
Commentary
12/17/2013
09:06 AM
Doug Henschen
Doug Henschen
Commentary
Connect Directly
Google+
LinkedIn
Twitter
RSS
E-Mail
50%
50%

Real-Time Prediction, Big Data Scale: Impossible Dream?

WibiData challenges IBM, Oracle, and SAS by combining vast customer history as well as real-time behavioral data to generate personalized recommendations.

Marketers have been talking about one-to-one marketing for decades, and the challenge is that much harder now that we can tap into big data.

Pioneering startup WibiData says it's bringing big retailers and financial institutions the best of both worlds, combining vast data stores from Hadoop and HBase with the very latest customer behaviors captured on websites and mobile apps. Best of all, application developers don't have to become data scientists to take advantage of the real-time recommendations.

"Front-end Web developers shouldn't have to think about how a predictive model is constructed, calculated, or applied," says Christophe Bisciglia, CEO and co-founder of WibiData and a co-founder of Cloudera. "They should just be able to get recommendations, take the results, and apply them however they want."

WibiData's differentiator is bringing both big data and real-time data into individualized recommendations for specific customers (not customer segments) as they interact with websites, mobile apps, in-store kiosks, or any other channel. The product behind that promise is WibiEnterprise 3.0, which was released late last month.

[Want more on next-generation demands? Read 5 Big Wishes For Big Data Deployments.]

WibiData's competition includes the likes of Oracle RTD (Real-Time Decisions); web-based services like Certona, iGoDigital, and Strands Recommender; and home-grown systems. But these options invariably can't handle big data -- such as log files or customer web or mobile clickstreams -- or they can't individualize the recommendation based on the latest browsing experience, according to Bisciglia.

RTD and online recommendation engines are "great for helping you understand that customers who bought X also bought Y," Bisciglia says, "but that doesn't capture the intent that the customer is expressing in that current session. It won't tell you that somebody is shopping for a gift, not buying what they normally buy. And it won't tell you that the customer just purchased a TV, so stop showing them other TVs and start showing them HDMI cables and speaker systems."

WibiData counts big retailers and global financial services among its customers, but these customers won't let their names be used, according to Bisciglia. Among the handful of WibiData customers talking about their deployments publicly is Opower, a company that analyses utility smart-meter data in real time to help customers understand and reduce their power consumption.

"Trying to solve big data challenges by combining [Hadoop and HBase] with traditional application development processes mires even the most talented developers and data scientists in tedious and laborious modeling and development cycles,"said Rick McPhee, Opower's SVP of Engineering, in a recent statement. "After facing some of these problems first hand, we’ve worked with WibiData to simplify and speed our big data application development processes."

WibiData has been working on the problem since its founding in 2010, but it has morphed along the way from providing a monolithic, proprietary product to offering enterprise support for modularized, open-source software. The Apache-licensed software is the Kiji Project framework, which works on top of Hadoop and HBase and includes two main components. Kiji Express is an analytical language in which you can build models or integrate with models developed in other languages, such as SAS or SPSS, using PMML (predictive model markup language). The models are executed in the Kiji Scoring Server, which has a REST interface that lets developers plug the models into web and mobile applications without having to know a thing about predictive modeling.

There are certainly others on the list of vendors working on the combination of big data and real-time, predictive analytics -- IBM, Pivotal, SAS, Teradata -- but Bisciglia claims cutting-edge data scientists are turning to a new generation of vendors to make it happen.

"This new breed of data scientists tends to use tools like R and Python and Scala," he observes. "We're not seeing them use the traditional tools and we're not talking about doing SQL querying on top of Hadoop."

Associating SQL with BI, Bisciglia says WibiData Enterprise customers can use the product's Hive adapter to query, say, how many times a customer clicked on an offer. But to build a predictive model that will present that customer with the right offer based on their history and their current session behavior? For that, WibiData's prescription is the Scala-based Kiji language to develop the model, its schema-management and model lifecycle-management tools, and its REST interfaces to capture real-time information and to plug the model into web or mobile applications. 

With giants like IBM, Oracle, SAS and others pursuing real-time e-commerce at big-data scale, it might be hard to think about a little startup with a funny name like WibiData as much of a threat. But Bisciglia knows all too well that that's what they said about Hadoop and Cloudera three or four years ago.

Doug Henschen is executive editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data, and analytics. He previously served as editor-in-chief of Intelligent Enterprise, editor-in-chief of Transform Magazine, and executive editor at DM News.

IT groups need data analytics software that's visual and accessible. Vendors are getting the message. Also in the State Of Analytics issue of InformationWeek: SAP CEO envisions a younger, greener, cloudier company (free registration required).

 

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Tullio Siragusa
50%
50%
Tullio Siragusa,
User Rank: Apprentice
12/17/2013 | 4:38:00 PM
Re: There are plenty of other options...
It is virtually impossible today to segment all the customer data into a one:one relationship "real time". It is also unmanageable especially in banking with so many data silos.  The idea behind SegONE is to do it near real time at "the touch point" with the customer - be it a visit to a branch, a call to a customer service center, and online transaction, or invoking a programmatic buy via an agency trading desk to present an ad to a consumer that is shopping for a car without it being creepy, that reminds them they are a customer and an offer that is relevant to them is available.

The question to answer about your scenario is why did they just buy a TV?  The transaction data only shows behavior after the fact.  The perpetually connected consumer data can tie in the why. Did the person just get a promotion a month before?  When was the last purchase they made for a TV?  When you triangulate both the context for what may trigger the purchase (got a promotion), with the history of transactions (is the customer due to upgrade now that they are making more money?) - you have context you did not have before.  This is obviously a very elementary view into the semantics of both sets of data that can help a bank gain knowledge on how to engage in relevant ways with each individual customer such as offering a bigger credit line, or loan, etc. 
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
12/17/2013 | 4:29:01 PM
Re: There are plenty of other options...
Tullio,

Thanks for the comment. "Micro-contextual segmentation consumer dossier at the touch point within the existing CRM workflow..." is quite a mouthful, but what is it? Is that synonymous with one-to-one-predictive offers? Wibi's claim is that's it's getting away from segments and predicting the right offer for individual customers. The latest interaction info is even more relevant, because if the offer is for a TV, but they just bought one, what good it the prediction?
Tullio Siragusa
50%
50%
Tullio Siragusa,
User Rank: Apprentice
12/17/2013 | 4:11:34 PM
Re: There are plenty of other options...
A framework to develop apps (wibidata) may help financial services firms do it themselves. An enterprise big-data vertical software application providing a micro-contextual segmentation consumer dossier at the touch point within the existing CRM workflow, tying in both structured/unstructured enterprise and perpetually connected consumer data semantically is underway by SegONE Inc.
Laurianne
50%
50%
Laurianne,
User Rank: Author
12/17/2013 | 12:50:40 PM
Real-time shopping intent
I always smile when i log into Amazon and it suggests "a little something for you," except it is based on gift purchases I have made for boys under the age of 10. So WibiData promises to figure out I am shopping for a gift, not for myself? What are the clues?
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
12/17/2013 | 11:05:22 AM
There are plenty of other options...
Beyond all the well-known vendors mentioned above, plenty of integrators and other startups are working on combining big-data analytsis and real-time prediction. WibiData is working toward productizing this capability and making it less of a one-off, reainvent the wheel proposition. It still requires plenty of development work on the app side and data science on the analytics side, but at least you're not starting entirely from scratch. 
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Dec. 9, 2014
Apps will make or break the tablet as a work device, but don't shortchange critical factors related to hardware, security, peripherals, and integration.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of December 14, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.