Data warehousing vendor Teradata on Wednesday announced a partnership with storage specialist RainStor aimed at affordable long-term data retention. The partners said their offering, to be introduced in the first half of 2011, will offer highly scalable on-line storage at costs that will be competitive with off-line tape.
The RainStor partnership is a response to growing demands to analyze ever-increasing volumes of data. Even as Teradata and its competitors have scaled out their platforms into the petabyte range, telecos, financial services, Internet marketers and other big-data generators are demanding longer-term storage options.
Storing and analyzing data for as long as three years is now a routine cost of doing business in many industries -- one that organizations gladly bear in exchange for analytic benefits including spotting market opportunities, reducing risk, optimizing profits and retaining best customers.
But the benefits may not outweigh the costs as data retention periods grow longer, driven largely by regulation. Governments around the globe are demanding that telcos store call records for longer periods. Financial regulators now want more detail and history on every trade, tick and price move on every exchange.
There are also extreme-scale scenarios involving network-log and RFID data, and extreme-longevity requirements for insurance and patient healthcare records.
RainStor tackles the retention challenge with a SQL-queryable database that uses value- and pattern-de-duplication techniques to deliver 50X compression. Teradata and competitors including IBM Netezza and EMC Greenplum currently support 3X to 4X compression. Oracle touts 10X hybrid columnar compression on its Exadata appliance. Column-store databases such as Sybase IQ, ParAccel and Vertica offer up to 100X compression.
Unlike these analytic products, RainStor's database can query data in its compressed format. That eliminates the un-compress step and improves performance, but the emphasis is low-cost retention and retrieval rather than fast analysis. The database is also said to require no special indexing or administration.
"You don't need a DBA to design and maintain the database; you load the data and it works," said John Bantleman, CEO of RainStor.
The joint product will be built on Teradata hardware and will provide a hierarchical option to migrate aging data to lower-cost, yet SQL-queryable on-line storage. Administrators will be able to set up retention policies, and access will be seamless to Teradata users, according to the vendors. Exact specifications and costs have yet to be determined.
"We're trying to position this as being as cost-effective as tape without compression; but the problem with tape storage is that it's off line," said Chris Twogood, senior director of product and services marketing at Teradata. "This will enable us to bring information on line affordably, so our customers can do fast SQL retrieval to meet compliance demands."
There's little doubt that competitors will respond. EMC, in particular, has opened up storage options as a new competitive front in the analytic data warehouse market. Its recently introduced EMC Greenplum Data Computing Appliance integrates with EMC DataDomain, a secondary storage and backup/recovery appliance, and EMC RecoveryPoint, software for replicating data onto storage area networks. Neither of these options addresses the low-cost retention challenge answered by RainStor, but EMC has other storage technologies it can draw on.
RainStor sells its technology exclusively through OEM partners. Hewlett-Packard uses RainStor technology in a product aimed at Telecom record retention and retrieval. Informatica uses the firms technology as part of a legacy-application data migration product. Current RainStor customers are said to include blue-chip companies such as Vodafone, AT&T, Fidelity and Lloyds Bank.
