2 Ways To Tackle Really Big Data

Infobright is not as big or well known as any of the competitors discussed above, but it has been around since 2008 and the 4.0 release of the Infobright database announced last week steps up capabilities specifically aimed at machine-generated data.

Infobright is not an appliance vendor and its database is designed to run on symmetric multiprocessor (SMP) servers rather than the massively parallel processing (MPP) approach used by EMC, IBM, Teradata, and others. The approach is all about fast query speed, but most deployments are in the 1-to-10 terabyte range and top out at about 40 terabytes (because you can't scale out as you can in the MPP approach). By exploiting links to Hadoop, however, Infobright says it can help companies affordably address both high-speed and high-scale.

Infobright gets its speed from its column-oriented architecture, which enables it to query selected attributes without wading through all the non-relevant data in each row that row-oriented databases (like EMC Greenplum, IBM DB2, Netezza, Oracle, Teradata) have to churn through.

Column-oriented databases also excel at compression, particularly when dealing with highly repetitive machine-generated data. Where row-oriented databases average 2X to 4X compression, Infobright says it routinely gets 10X compression and peaks at 40X compression. That makes big data smaller, saving money on processing power and storage.

The Infobright 4.0 adds two features expressly for machine-generated data. First, a DomainExpert feature lets companies store repeating patterns of data that don't change, such as email addresses, URLs and IP addresses. These three examples are included in a Web analytics DomainExpert set developed by Infobright and included in 4.0. But companies can add their own patterns as well, whether it's related to call data records, financial trading, or geospatial information. The query engine then has the brains to ignore this static data and instead query only the changing data. That saves query time because irrelevant data doesn't have to be decompressed and interrogated.

The second new feature that speeds analysis is Rough Query for Data Mining, which is an approach whereby in-memory metadata about each column and row is queried first to eliminate all data that's not relevant to the query. Once the relevant information is revealed, the query engine issues a select statement query only against that relevant data. It can speed queries by as much as 20 times, according to Infobright, over the conventional approach of issuing a long-running query against the entire data set.

Infobright says customers such as hedge funds investigating price histories, financial firms doing portfolio risk analysis, marketing organizations doing clickstream analysis, and so on all want the ultimate in fast querying. If those customers also have large-scale archiving needs, Hadoop, the fast-growing open source project, includes options for low-cost, queryable storage. Like many commercial vendors, Infobright has integrated its product with Hadoop, so data stored there--or subsets of data boiled down through Hadoop processing--could be brought back in Infobright for fast SQL-style analysis.

The downside of using a SQL database (like Infobright) and Hadoop in combination is that you can't use the same SQL-based queries and applications in the latter environment--an advantage IBM points out in highlighting the advantages of its High Capacity appliance. On the other hand, Hadoop deployments running on commodity hardware can cost as little as $250 per terabyte, according to Cloudera, which provides commercial service and support for Hadoop deployments. That's quite a savings over the less-than-$2,500 per terabyte touted by IBM.

So there you have the two approaches to handling machine-generated-data. If you have vast archives, EMC, IBM Netezza, and Teradata all have purpose-build appliances that scale into the petabytes. You also could use Hadoop, which promises much lower cost, but you'll have to develop separate processes and applications for that environment. You'll also have to establish or outsource expertise on Hadoop deployment, management, and data processing.

For fast-query needs, EMC, IBM Netezza, and Teradata all have fast, standard appliances and faster, high-performance appliances (and companies including Kognitio and Oracle have similar configuration choices). Column-oriented database and appliance vendors including HP Vertica, Infobright, ParAccel, and Sybase have speed advantages inherent in their database architectures.

As always, your performance will vary depending on your queries, your data, data volumes, query volumes, number of users, and other factors. Do thorough tests with your own data and your toughest queries to determine which path to follow.

You can't afford to keep operating without redundancy for critical systems--but business units must prioritize before IT begins implementation. Also in the new, all-digital InformationWeek SMB supplement: Avoid the direct-attached storage trap. Download it now. (Free registration required.)