Teoco Discovers the Good, the Hidden Costs and the Challenges of Data Warehouse Appliances

What are the pros and cons of using a data warehouse appliance versus a more traditional data warehouse? We asked John Devolites, general manager, communications and entertainment solutions at Teoco, to share with us his experiences with three data warehouse appliances. The bottom line: he's very happy with the price and his new, fast query rates, but don't call these appliances -- they're a far cry from the simple, plug-and-play boxes that the word "appliance" conjures up.Teoco, The Employee-Owned Company, provides software used to process and audit billing among 40 telecommunications-carrier clients -- in one case, handling a billion invoices a month for just one carrier. Last year, the company started running into run-time and report processing issues with its Oracle-based data warehouse (Teoco continues to use an Oracle-based transaction processing system). "If you were a telephone company and it was taking you 20 hours to run a financial report at the month-end close, that would be a significant issue," Devolites points out.

Telecommunications company customers -- one of which had recently merged, doubling its monthly transactions -- urged Teoco to get a new data warehouse. The company obligingly looked at three products, seeking one that could run 50 to 100 times faster than the existing Oracle system and that had a good price. In a benchmark test of the Datallegro C25 (which holds up to 25 terabytes), Teoco loaded five terabytes into the system and immediately started seeing that desired 50- to 100-fold improvement in query response time.

So far, the company has bought one Datallegro data warehouse appliance and two from Netezza. A query that once took Oracle 20-22 hours to run now takes minutes.

Some have argued that in a growing and rapidly changing business, the last thing you want is proprietary, single-use hardware. Devolites strongly counters this notion. He points out that the Datallegro system is based on Dell and EMC hardware with an Ingres open source operating system and that Teoco runs several software programs on top of the appliances, something you couldn't do with proprietary hardware. In fact, he thinks the word "appliance" is a misnomer. "Vendors have tried to say it's an appliance to give you the impression it's like a refrigerator and all you do is plug it in," he says. "It's not. It's a massive parallel processing computer system with an open source operating system on it that we have to write code to work with." In fact, he estimates Teoco has spent $10 to $15 million on writing such code, to perform data enrichment and error checking and to load data into the appliances. With traditional data warehouses, extract, transform and load happen in that order. With appliances, it tends to be extract, load, and then transform the data on the target box. "What we spend all our time on in engineering is getting the data enriched correctly before it goes into the appliance for use," Devolites says. "Then we have sophisticated algorithms on the appliances themselves that look for patterns in the data. Then our users, typically employees in the telecommunications companies, use Business Objects to run their own reports, do their own queries, and create their own graphs."

Even with the expensive coding required, Devolites says data warehouse appliances offer a cheaper per-terabyte price than traditional data warehouses. "I'd have to write code for anybody's box," he says.

In addition to faster reports, the data warehouse appliances offer Teoco the ability to analyze data it couldn't keep up with before. It can look more closely at long-distance call records, for instance, and detect end-office patterns, call routing patterns, roaming fraud, cloning fraud (when someone hops on the network spoofing another phone), end-user fraud, and marketing opportunities.

For one carrier, Teoco's team of fraud auditors ran a list of the top 100 users and found ten who were using 25,000 minutes a month. Upon closer inspection, it turned out that a cab company was using a consumer wireless phone plan for dispatch, taking advantage of the plan's free nights and weekends. This didn't require any sophisticated analysis to figure out, Devolites notes, but it did require the right tool to dig it out. Teoco runs Business Objects business intelligence software on all its data warehouse appliances and SPSS's Clementine predictive modeling on the Netezza boxes.

One problem to which Teoco is targeting its analytics is "phantom traffic" encountered by traditional wire line companies, wherein the traffic they're hauling isn't properly rated and processed. Teoco's analysis checks to make sure the right rates are being applied to inter-carrier bills. Some top carriers do $2.5 billion to $3 billion worth of activity in this area, so finding even a half-percent error is worthwhile.

Future applications of analytics could lead to interesting new opportunities for Teoco. For instance, in the U.S. in 2006, $500 million worth of ringtones were downloaded. People are increasingly buying more music, songs, ringtones, and video, which makes subscriber analytics more interesting. A wireless company could apply predictive analytics to its customer list and offer predictions to a music company about which of its customers are likely to buy music from a particular artist.What are the pros and cons of using a data warehouse appliance versus a more traditional data warehouse? We asked John Devolites, general manager, communications and entertainment solutions at Teoco, to share with us his experiences with three data warehouse appliances. The bottom line: he's very happy with the price and his new, fast query rates, but don't call these appliances -- they're a far cry from the simple, plug-and-play boxes that the word "appliance" conjures up.