Nine BI Megatrends for 2009

The world may be flat, as author and New York Times columnist Thomas Friedman has observed: but these days, it feels more like a deflated soufflé. Great sums of real and imaginary wealth blew out of the economy like overheated gas, leaving behind a sticky, gooey mess. Financial crises have been swirling about the globe, sucking the life out of commerce, launching a global recession and exposing weakness in nearly every business model. Sophisticated use of information and the deployment of advanced algorithms and software processes clearly helped organizations uncover opportunities and develop information-based products when things were going up. But apparently too few employed their wealth of knowledge to avoiding the downsides of risk. Now, organizations will need many of the same sorts of tools to scrutinize processes and decisions that exposed them to their current troubles and discover how to survive and prosper in a remade economic world.

During such difficult and rapidly changing times, it would be hard to argue against the importance of business intelligence for providing executives, managers and front-line knowledge workers with the tools they need to use information effectively. While it has always been tough to measure the exact impact of BI and data warehousing on overall business performance, it is certain that in many organizations, implementation of BI and data warehouse tools and technologies has forever changed how users access, analyze, report and share data. Excellence in these activities is for many organizations as important as transaction or process efficiency, and essential to the cycle of improvement. In other words, no matter how bad things get, it is unlikely that organizations will turn away from BI and go back to more primitive tools and methods.

Yet, there is also little doubt that BI and data warehousing providers and professionals will have to adjust to a deflated world and refashioned expectations and objectives. There will be less patience with delays in the development and deployment of systems and promised returns on investment. The margin for error will be tighter, with fewer second chances. CIOs, CTOs and business analysts will be under pressure to manage costs associated with information analysis and reporting.

With the omnipresent economic challenges serving as backdrop, what are this year's BI megatrends? Which BI technology developments are most important to your organization? This article describes nine important trends that are shaping the future of business intelligence and information management, from open source software and embedded BI to analysis of data relationships and business models to data warehousing alternatives including MapReduce and column-store databases. But first, let's consider what happened with last year's top megatrend.

Consolidation Conundrums

Twelve months ago, the uber-megatrend was industry consolidation. IBM, Oracle and SAP had just gobbled up Cognos, Hyperion and Business Objects respectively; together with Microsoft, these vendors seemed poised to bring down the curtain on the old BI industry and begin a new phase dominated by enterprise application and information management technology heavyweights.

As it turned out, in 2008 these four giants made significant technology moves but spent a good deal of their time under the hood, retooling their newly integrated and sometimes overlapping product, marketing and sales strategies. As expected, they have positioned BI as part of something they offer that's bigger; in fact, the term "business intelligence" doesn't get much respect anymore. IBM, for example, positions BI as part of a larger thrust toward "business optimization," which it sees as different from mere application automation. Optimization is about using information as a strategic asset to improve business processes, performance management, models, forecasts and policies; it can involve a range of technologies, not just traditional BI tools for data access, reporting and analysis. Oracle and SAP take a less visionary approach, offering the tools as items in their middleware or solutions portfolios (although Business Objects, as an SAP company, still has high visibility). You have to do a bit of searching on Microsoft's site to find information about BI, but that could be because it takes a village; Microsoft's Business Division, Data and Storage Platform Division and the Office Business Platform Group all contribute technologies that BI developers, IT managers and business users are seeking.

The internal machinations of the big four have left room for independents, including Actuate, Information Builders, Jaspersoft, MicroStrategy, Pentaho, SAS, TIBCO Spotfire and others in analytics, performance management, search and vertical industry areas. On the data warehouse systems and middleware side, IBM, Informatica, Oracle and Teradata come into 2009 as heavyweights. Not much was heard in 2008 from Hewlett-Packard's NeoView, but HP made its presence felt through partnerships, particularly with Oracle to produce a data warehouse appliance. Innovation in information management is being driven by increasing customer demand for more timely and diverse data, faster query performance, better data quality, master data management (MDM) and more. Appliances, column-oriented databases and other new approaches are providing organizations with more options and are sending a message to the heavyweights that complacency equals vulnerability. Even Teradata has loosened up its enterprise data warehouse dogma to bring to market an appliance that fits the price and performance requirements for departmental data marts.

Megatrend 1: The Impact of Open Source

Some commercial vendors might dismiss it, but open source is a growing factor in BI, data integration and data warehousing, and is clearly worthy to be cited as the first megatrend. The potential for lower total cost of ownership (TCO) is certainly attractive, but there are other important reasons that the open source BI movement is gaining momentum. Many organizations and developers are ready to build up from generally positive, maturing experiences with systems constructed using the LAMP stack (Linux, Apache, MySQL and PHP, Perl or Python) and the Eclipse development platform. Others like having more control over the code and direct involvement in the process of updating the software; these factors are important for organizations that prefer incremental deployment where users test components, provide feedback and revise requirements along the way rather than wait for finished systems. This is critical because in many organizations, BI deployments are delayed when user requirements have not been met or have changed during the time frame of the development cycle.

New vendors are drawn to open source because it offers a lower-cost, contemporary marketing model. Working with communities, partners, influential developer forums, social networking sites and bloggers, these vendors are executing marketing plans that creatively – but not too aggressively – build awareness and try to increase download rates. Coming on too strong can have negative repercussions, so wiser vendors calibrate their marketing to build confidence so that downloaders of free software become buyers of feature-rich, enterprise versions. Vendors also have to alleviate concerns that open source products will hit a ceiling as organizations scale up. Buoyed in most cases by infusions of venture capital, open source BI vendors have been developing their products rapidly. Pentaho's BI Suite Enterprise Edition, for example, offers features that are comparable to commercial BI products. Actuate, which has its roots in the commercial software market, based its current major release (Actuate 10) on the Eclipse Business Intelligence Reporting Tool (BIRT) project.

The most prominent open source data integration vendor is Talend; several open source BI and data warehouse vendors resell the company's products. The ability to customize data integration middleware through access to the source code is attractive because like it or not, many organizations need to tailor such routines to their requirements. Out-of-the-box routines only go so far. Cost is also a critical factor, given that commercial software and services for data integration have traditionally been one of the more expensive parts of the BI and data warehouse stack.

Finally, thanks to open source, database management for BI, analytics and data warehousing are undergoing something of a renaissance. Kickfire, Infobright and others are focusing particularly on the needs of maturing MySQL shops and are using aspects of the open source model to gain traction. In sum, the open source style of marketing products and services is changing how nearly all vendors – not just open source ones – engage with customers and introduce new technology.

Megatrends 2, 3 & 4: BI Tool Innovation

With the economy in turmoil, the last thing users need are BI tools that lock them into older ways of accessing, analyzing, reporting and sharing information. Search engines, mashups and social networking have changed the way most people interact with information and have raised the bar for BI tools to provide a similarly rich experience in a business context. Of course, therein lies the rub; business requirements are tougher than typical searching and sharing. Nonetheless, vendors are making progress in modernizing the BI experience, certainly compared to its roots in batch querying, canned reporting and analytics that once required hand-holding by PhDs. Here are three BI tool megatrends to track in 2009:

BI becomes less isolated. As analysis, information access and reporting become integral to operational decisions and actions, BI tools and methods cannot live in a silo. A recent Ventana Research "Operational BI benchmark study found that most operational users currently employ reporting, access and analysis tools that come with functional applications rather than dedicated BI suites. While the dedicated suites can help break down silos, these deployments cannot simply become bigger BI silos; they have to be well integrated with, if not embedded in applications, services and processes that are important to operational users.

One frequently overlooked need is the integration of BI and performance management with project management. Managers need timely information and analytical capabilities to help them track multiple projects and evaluate with data whether ongoing projects are meeting strategic and budget objectives. KnowledgeRelay is an innovator in this area; its BI and performance management products and services have evolved out of experience in aerospace, defense and other industries known for having numerous interlocking and complex projects. From the project management side, Planview, a leading project portfolio management (PPM) vendor, beefed up the BI components of its Planview Enterprise suite in 2008.

Users demand a richer experience. Anyone looking at the latest BI user interfaces can see that the days of simple, canned reporting supplemented by ad hoc querying functions are slipping into the past. When BI first began migrating to Web browsers, functionality seemed to take a step backward. But now, BI dashboards and portals are on a path to become richer and more flexible. This is in large part due to their implementation of rich Internet application (RIA) tools and methods such as Adobe Flash. Actuate's Actuate 10 offers the most fully developed merger of RIA with BI, but other vendors are also delivering rich and dynamic BI interfaces.

On the R&D drawing boards of many vendors are tools to create mashups, allowing users to combine different data sources and visualizations in a self-service manner rather than having to rely on IT developers. Of course, mashups may require some users to jump through not just technological but also political, managerial and security hoops regarding what can appear on shared BI dashboards and portals. However, the richer interfaces have the potential to enable users to integrate "at the glass" a wider variety of data sources, such as from geographic information systems, external demographic data and more. Look for search, wikis and blogs to also play starring roles in BI dashboards and portals.

BI will focus on relationships. "Connect the dots" has become a mantra in government intelligence circles, but it is important for other objectives, such as customer experience management, risk and security management, Web data analysis, supply chain management and business performance management. For most of its history, BI has focused on reports that do not provide much flexibility for investigating data relationships, leaving this analysis to specialists using data mining or statistical analysis tools. But as BI becomes integral to operational decision making, users will need to examine real-time data, text and alerts in a context that improves understanding of relationships and dependencies. This requirement will increase the need for master data (or data relationship) management tools at the integration layer, but BI and business user analytics tools will also play a critical role. TIBCO Spotfire's DecisionSite and Analytics Server are pacesetters, with recent releases enabling organizations to visually analyze data relationships, including, for example, those created during drug tests or by real-time interactions between participants in social networks. And coming from a data-integration middleware direction, Composite Software introduced a new product, called Composite Discovery, which uses search, metadata and other techniques to discover and display data relationships.

Megatrend 5: Business Modeling Meets MDM

BI must support requirements for greater business agility at a time of rapid changes in the economy. Tools that offer self-service information access, analysis and reporting are critical, but users need more. They also need the ability to build and revise business models in ways that can be easily mapped to underlying data models. Business analysts and users have traditionally had a difficult time understanding data models, which can be complicated, hard to visualize and difficult to understand because they are often expressed in a specialized language or methodology. Important requirements can be lost in translation, leading to delays in the deployment of BI and data warehouse systems when functions must be added or significantly modified at later stages.

The megatrend here is about MDM and emerging semantic models, which could serve business modeling in the same way that data models, schema and metadata have served extract, transformation and loading (ETL) tools. However, MDM and business modeling must be integrated with underlying data models. Some vendors in the MDM market, looking for ways to appeal more directly to potential business user customers, are developing tools to improve business modeling and its relationship to data modeling. Kalido is at the front of the pack; in 2008, the company introduced Business Information Modeler, a graphical tool for business users and analysts to develop conceptual models that can more accurately represent information needs and business processes. Another vendor to watch is Expressor, led by Bob Potter, who was previously CEO at Kalido. Expressor's Semantic Data Integration System offers a set of component products that focus on the integration of business modeling with data transformation and business rules.

IBM Cognos is also addressing these needs with Business Viewpoint, which it introduced as part of IBM Cognos 8 version 4; integrated with IBM's InfoSphere MDM Server, the technology enables organizations to align business modeling with MDM. An important element of the Cognos offering is dimension management, which helps organizations improve their governance and maintenance of hierarchies and dimensions. These tasks prove difficult as the number of users, the size of the artifacts and requirements for refreshing the data increase. Dimension management is also featured in MDM products from Microsoft and Oracle, but IBM Cognos is ahead in enabling business users to have access to this management functionality in the BI platform.

Megatrends 6, 7 & 8: Breaking the BI/DW Mold

Pressures to provide more timely and comprehensive understanding of business performance, customer behavior, supply chain management, regulatory compliance and more are driving significant changes in BI and data warehouse tools, systems and architecture. The result has been one of the more creative periods in technology for analytics and information management in recent memory. Some innovations employ very large memory and near-line storage to provide faster querying and greater data availability; others achieve results through new programming and event processing techniques. Most of these innovations take advantage of hardware advances in parallelism, virtualization and blade servers to enable higher performance at a reasonable cost. Here are three megatrends to watch:

MapReduce meets large-scale data analysis. Implemented most famously by Google and Yahoo but also available in the open source Apache Hadoop framework for data-intensive distributed applications, the MapReduce programming technique is suddenly the rage. MapReduce is growing in importance as organizations begin to build parallel, virtualized architectures based on server farms using commodity hardware, in many cases to support anticipated growth in cloud computing, or software-as-a-service (SaaS). Not everyone is on board with MapReduce; database luminaries Michael Stonebraker and David DeWitt argue that it is a step backward (see "The Database Column" ). Nonetheless, data warehouse appliance vendor Greenplum and analytical database provider Aster Data Systems are the first out of the box with MapReduce technology. More are sure to follow.

Column-oriented databases take aim at performance woes. In the recent Ventana Research "Optimizing BI and Data Warehouse Performance" research study, more than half (58%) of participants said they are experiencing sometimes nightmarish performance problems when they have to scale to run more complex queries, and nearly half (48%) said they have the same problem when scaling to load more data. As a result, they are canceling important queries when they simply run too long. Thus, it's not surprising that the study found that organizations are evaluating appliances and column-oriented databases to remedy problems in these areas.

Sybase IQ is the established column-oriented database system; Infobright, ParAccel, and Vertica are leading contenders. In addition to column orientation, these products employ advanced compression technology and large memory to change the game for BI and data warehouse architecture. It's fair to expect that in 2009, database heavyweights IBM, Oracle and Microsoft will continue to make moves to increase their presence in the market for appliances and perhaps even column-oriented databases.

Event processing opens new analytical possibilities. Before the financial services industry cratered, that was where most of the work in event or stream processing could be found. Now, while algorithmic trading and other processes still consume this technology, the spotlight shines brighter on emergent applications in healthcare, telecommunications, government intelligence, IT management, gaming and Web analytics. Network events, sensor data from radio frequency identification (RFID) tags and surveillance data are among the new sources. Capturing events, correlating them and presenting the results of analytics in dashboards can potentially give organizations more actionable insight than traditional BI tools provide. However, to gain full business value, event processing must be deployed in an integrated fashion with not only BI and data warehouse systems but also process management and service-oriented architecture. IBM, which acquired AptSoft in 2008, also offers Cognos Now (formerly Celequest) and is flexing its muscles. Coral8, Progress Software (Apama), Streambase, TIBCO and Vitria are also important technology providers.

Megatrend 9: Too Big to Fail

Queries running against multi-billion row tables; data warehouses managing hundreds of terabytes and adding more every day; thousands of users dependent on information and analysis for daily operational decisions: Clearly, BI is something that organizations can't afford to take lightly. Yet, many organizations fly blind when it comes to managing workloads and setting priorities for improving performance. Research finds that while many IT database managers meet regularly with business users to solve performance problems, they often lack hard information to aid decisions about tuning existing systems and deploying new technology. Thus, a final megatrend is that we will see more implementation of BI to improve BI. Appfluent is a leading vendor of workload analysis and tuning tools that are being applied to BI performance management; Teleran is another.

Tighter budgets and a critical need for better alignment between business priorities and technology decisions mean that BI and data warehouse systems must be effectively and efficiently managed. New technology deployments and upgrades to existing systems must be carefully chosen to provide the greatest return on investment. Fortunately, thanks to a torrent of innovation in recent years, organizations have many technology options in front of them. While data security and availability concerns must be addressed, organizations are likely to see SaaS and cloud computing alternatives as part of the mix for BI, analytics and data warehousing as we progress into 2009.

The megatrends presented in this article show that BI is hardly a settled domain; there's too much at stake for organizations to stand pat when it comes to using information to improve decisions and take action. Failure is not an option when it comes to BI. It will take smarts to ride out a tough economy and identify where opportunities for growth lie. Fortunately, the BI market is responding to demand.

David Stodder is an analyst and writer focused on business intelligence, analytics and information management. He is a research fellow with Ventana Research, where he was previously vice president and research director with a concentration on information management, operational intelligence and IT performance management. He was the founding chief editor of Intelligent Enterprise. He can be reached at [email protected].