Megatrends 6, 7 & 8: Breaking the BI/DW Mold

Pressures to provide more timely and comprehensive understanding of business performance, customer behavior, supply chain management, regulatory compliance and more are driving significant changes in BI and data warehouse tools, systems and architecture. The result has been one of the more creative periods in technology for analytics and information management in recent memory. Some innovations employ very large memory and near-line storage to provide faster querying and greater data availability; others achieve results through new programming and event processing techniques. Most of these innovations take advantage of hardware advances in parallelism, virtualization and blade servers to enable higher performance at a reasonable cost. Here are three megatrends to watch:

MapReduce meets large-scale data analysis. Implemented most famously by Google and Yahoo but also available in the open source Apache Hadoop framework for data-intensive distributed applications, the MapReduce programming technique is suddenly the rage. MapReduce is growing in importance as organizations begin to build parallel, virtualized architectures based on server farms using commodity hardware, in many cases to support anticipated growth in cloud computing, or software-as-a-service (SaaS). Not everyone is on board with MapReduce; database luminaries Michael Stonebraker and David DeWitt argue that it is a step backward (see "The Database Column" ). Nonetheless, data warehouse appliance vendor Greenplum and analytical database provider Aster Data Systems are the first out of the box with MapReduce technology. More are sure to follow.

Column-oriented databases take aim at performance woes. In the recent Ventana Research "Optimizing BI and Data Warehouse Performance" research study, more than half (58%) of participants said they are experiencing sometimes nightmarish performance problems when they have to scale to run more complex queries, and nearly half (48%) said they have the same problem when scaling to load more data. As a result, they are canceling important queries when they simply run too long. Thus, it's not surprising that the study found that organizations are evaluating appliances and column-oriented databases to remedy problems in these areas.

Sybase IQ is the established column-oriented database system; Infobright, ParAccel, and Vertica are leading contenders. In addition to column orientation, these products employ advanced compression technology and large memory to change the game for BI and data warehouse architecture. It's fair to expect that in 2009, database heavyweights IBM, Oracle and Microsoft will continue to make moves to increase their presence in the market for appliances and perhaps even column-oriented databases.

Event processing opens new analytical possibilities. Before the financial services industry cratered, that was where most of the work in event or stream processing could be found. Now, while algorithmic trading and other processes still consume this technology, the spotlight shines brighter on emergent applications in healthcare, telecommunications, government intelligence, IT management, gaming and Web analytics. Network events, sensor data from radio frequency identification (RFID) tags and surveillance data are among the new sources. Capturing events, correlating them and presenting the results of analytics in dashboards can potentially give organizations more actionable insight than traditional BI tools provide. However, to gain full business value, event processing must be deployed in an integrated fashion with not only BI and data warehouse systems but also process management and service-oriented architecture. IBM, which acquired AptSoft in 2008, also offers Cognos Now (formerly Celequest) and is flexing its muscles. Coral8, Progress Software (Apama), Streambase, TIBCO and Vitria are also important technology providers.