With their data stores of 6 terabytes and 2.5 terabytes, respectively, LinkShare and Wealth Engine aren't going to win any big-data prizes. What these two marketing firms, among the latest customers to choose Oracle's Exadata Database Machine, need most is quick loading and fast querying.
Size isn't everything, and given that some 90% of enterprise data warehouses hold less than 10 terabytes of information, most practitioners are more concerned about requirements other than the ultimate in scalability.
In many marketing scenarios, for example, speed is everything. Marketers need to load and analyze data as quickly as possible so the insight can be quickly applied to identify new segments and lists and to improve the targeting or creative content for the next campaign. If you know what's working sooner, you won't waste money on segments, data or marketing appeals that aren't fruitful.
LinkShare Tracks the Clicks
LinkShare provides search, lead generation and affiliate marketing services to two constituencies: publishers of the sites that are registered to its network and the advertisers who make use of that network. Advertisers on the network typically pay publishers on a percent-of-completed-sale or cost-per-action basis, with retail merchandise and credit card offers being typical.
To help these constituents discover what's working, LinkShare has been moving toward self-service access to information. That initiative began in 2007 when the company upgraded its dashboard for the thousands of top publishers that are the biggest traffic drivers on its network. The dashboard lets analysts look for trends among top-producing advertisers on their sites, top link types, and high-performing campaigns.
By 2009, LinkShare's data warehouse had reached 4.5 terabytes of usable data, it was adding tens of gigabytes of new clickstream data per day, and it was encountering serious performance problems.
"We were getting to end-of-life stage on our incumbent data warehouse," says Johnathan Levine, chief operating officer and vice chief technology officer. "We had twice-a-week meetings just for tuning and we were spending huge amounts of time on tuning and [data] denormalization."
Levine wouldn't identify the incumbent database, but he says it had been in use for about six years. A Google search of past LinkShare coverage turned up several article references to a conventional DB2 database deployment in a clustered Linux environment.
If LinkShare's team of three DBAs and four ETL specialists could schedule downtime, they could load new data in about an hour. But if customers were online, querying the system, loads took as long as six hours, Levine says. LinkShare has large customer bases in both North America and in Japan, where its parent company, Rakuten, is headquartered. That global footprint made it difficult to schedule any downtime.
"We were trying to feed the data warehouse continuously because there was so much data that if we didn't load, we couldn't catch up," Levine says.
Industry performance benchmarks have also been raised, making low-latency insight a competitive expectation. "Five years ago it was okay to give people yesterday's data, but that's not good enough any longer," Levine says.
An upgrade was inevitable, as LinkShare wanted to do for its advertisers what it already done for publishing clients: provide a self-service dashboard, in this case for spotting top-producing sites by product category, campaign and offer.
After considering half a dozen vendors in late 2009, LinkShare settled on three candidates including its incumbent database. Oracle joined the fray "at the eleventh hour," as Levine describes it, when it was preparing to announce Exadata Version 2.
LinkShare settled on Exadata in part because it promised the ability to support continuous loading without bringing the database offline. LinkShare took delivery of the first of two half-rack clusters last March and it was operational by July (in a four-month deployment including a month of optimization work). The second half-rack was delivered in June and went live in September (a faster, three-month deployment).
LinkShare's ETL jobs still take about an hour to complete, but they can be done while this system is still online with thousands of customers actively querying the system. In fact, LinkShare has since launched a self-service dashboard for Advertisers, which has pushed the user base over 4,000 worldwide. Nonetheless, data is now typically no more than one hour old, whereas it ran a day or two behind in the previous system.
Query performance has also improved, according to Levine. The old target was to have 98% of queries returned within eight seconds, but the company was not consistently meeting that goal. Exadata met that benchmark out of the gate, so LinkShare set a new target of returning 98% of queries within four seconds. In January LinkShare reported that 96% of queries had met the four-second target and the company was optimizing to meet the 98% goal.
"The new level of performance we're delivering is quite important to our advertisers and publishers because four seconds is about the attention-span breaking point," Levine concludes. "It's now a much more interactive experience, which encourages people to ask a lot more questions and they're much more engaged with the application."
WealthEngine Builds A 'Data Factory'
Speed is as important to WealthEngine as it is to LinkShare, but for this data provider it's about quickly aggregating data from more than 35 different sources to help marketers and non-profit fundraisers find rich people. The two largest sources are state business registration records and a 180-million-household marketing list. To this, WealthEngine adds real estate records, donation databases, SEC records, boar registrations, social security information and lists from Dun & Bradstreet and other sources.
WealthEngine has more than 2,500 customer firms, and that figure is expected to grow by more than 50% this year. Customers buy data three different ways: they can use the WealthEngine Web site to look up names and addresses one at a time, or they can download targeted lists of up to 6,000 names. Larger lists with 100,000 to 5 million records are transmitted to direct marketers and advertisers.
The one thing all these customers want is fresh data, but that was getting to be a problem with WealthEngine's conventional Oracle 10g database deployment. According to Silas Matteson, senior vice president of products and IT, there were two catalysts of the new deployment: rapid data growth, with a doubling of the company's data store to 2.5 terabytes in 2010, and a desire to upgrade the company's services.
"We wanted to deliver a new user interface offering more aggregation and analysis of the data, but the old setup couldn't get us there," Matteson explains.
Daily updates, entailing both overwrites and appends to the database, were taking as long as 12 hours. But in tests of Exadata conducted at an Oracle facility in Reston, VA., WealthEngine found it could execute the updates in less than three minutes. (Oracle generally avoids on-site proof-of-concept tests, preferring to do lab-based benchmarks at one of nine test sites internationally.)
WealthEngine has yet to move into production and prove actual performance, but Matteson says he's satisfied with the test results.
"We used a copy of our production database, loaded a state business registration list and did joins and ad-hoc queries on two of our largest tables," he says.
WealthEngine took delivery of its quarter-rack Exadata Database Machine in January and testing is now underway. If all goes according to plan, WealthEngine will start using it by mid April as what Matteson calls a "data factory" that will continuously update four master tables: one on people, one on addresses, one on organizations and a cross-reference file cutting across all these sources.
Matteson says the deployment is not so much a data warehouse as a pre-aggregation engine that supports optimized searching across the company's many data sources.
"Exadata will enable us to load and aggregate data very quickly, and that, in and of itself, will give us a pay off because our data sources continue to grow," Matteson says.
WealthEngine's investment is not small. Matteson says it will cost about $900,000 for the hardware, database licenses, RAC (cluster) licenses and implementation services. And that's just the first phase. In a second phase envisioned for 2012, WealthEngine plans to implement a second quarter rack to support ad hoc querying.
"Our clients want to find prospects with certain attributes, such as age, education level, location, and affinities -- are they into horses, collecting art, NASCAR or boating?" Matteson explains.
WealthEngine has access to all that data, but it's currently limited in how flexibly you can combine searches of attributes. "When we tested Exadata, we saw first-hand how fast it could be in supporting ad-hoc search," he says.
A number of modern platforms might have delivered performance gains over the aging, underpowered databases in use at LinkShare and WealthEngine. But these deployments are not about big-data scale.
Lots of data warehousing and analytic platform providers talk up the demands of big-data environments these days (and we're still waiting to see an Exadata deployment north of 20 terabytes, let alone 100). But for every one practitioner stepping up into the hundreds of terabytes, there are roughly nine firms with equally tough problems that don't involve the ultimate in scalability.
For marketers in particular, speed is the first of three crucial attributes expressed in the age-old customer segmentation formula of recency, frequency and monetary value. Without speed, marketers can't see the most recent customer transactions and are, therefore, less likely to find best prospects.
As Levine of LinkShare explained, day-old data is no longer good enough, and WealthEngine faces the same pressure in delivering the very latest insight on high-net-worth marketing prospects.
As the database marketshare leader, Oracle will be called upon by existing customers to satisfy a greater diversity of deployment needs than is likely to be faced by any other vendor. At LinkShare it has passed the test of continuous data loading, and at WealthEngine it appears to be on its way to delivering a high-speed data aggregation platform.
I'm writing about IT's role in marketing's new digital age in the April 11 issue of InformationWeek. If you have fresh, noteworthy experience as a digital marketer (no marketing service firms or data providers, please), drop me a note.