BI Trends and Highlights Seen at TDWI

Every TDWI conference leaves me with insight into what to expect during the next year. At last month's World Conference in San Diego we had both the regular conference and a two-day executive summit. The main trends I saw:

Predictive Analytics still hot - Predictive analytics (a.k.a. data mining) seems to be the topic most people are interested in hearing about. A lot of the engineering problems we faced with data mining in the '90s have been solved and cheap computing power makes broader use more feasible. The catch is that it still takes expertise to understand which techniques work best for different problems. Expect a new subclass of BI professionals who know PA tools and techniques, just as we have BI tool and design experts now.

Lots of people talking about data governance - There seem to be two threads driving this:Data governance as a reaction to (a) regulation and requirements and (b) discomfort with the fact that a lot of what end users do is outside the control of central IT. I see a lot of IT people confusing (b) with (a) and either doing the right thing for the wrong reason or vice versa. It's going to be a long time until these issues are sorted out. In the meantime, I expect more alienated users.

Data warehouse databases and appliances - The "high end" data warehouse just keeps getting higher. Most people are dealing with 20 percent to 40 percent annual database growth rates. This caught people off guard who were expecting the ceiling on data volumes to stay linear. It's not. BI scalability and performance continue to be a problem despite continual hardware improvements.

This is fueling interest in all sorts of technologies. We have appliances like Netezza, DATAllegro, Dataupia. There are data warehouse database alternatives to the big three, like Teradata, ParAccel, Greenplum, Vertica, and Sybase IQ. There's renewed interest in accelerator products like HyperRoll or SAP's pricy BI Accelerator.

Open source BI interest rising - There's more awareness of open source alternatives for data warehousing tools. This year several commercial open source companies entered the market. You can now find open source reporting, BI, ETL and database software, and support contracts to go with them.

BI consulting companies popping up - The trade show had more small integration and consulting companies than I've ever seen at TDWI. Based on the number of these companies, the BI market has to be growing. It's also a reflection of the maturing software market. More complex tools, tool integration challenges, and new technology are all feeding the need for specialists.

Clickstream coming back? - I had a half-dozen conversations about clickstream data, up from the one I usually have. Web analytics has recovered from the bad days around 2001 when the bottom dropped out of that market.

What's changing is that many people who bought Web analytics packages or used hosted services are now facing the problem of how to marry web data with the rest of the data in the organization.

GIS, location intelligence, geocoding - call it what you want. Based on attendee interest, it's going to resurface in the near future. There was a point a while back where this was in the hype cycle but not really ready for broad deployment. Since then there's been a lot of engineering work done, making it easier. It's an area to watch. In the meantime you can play with Google Maps mashups.

The Web 2.0 dark horse - The impact of the new Web technologies, companies, architectures and expectations is visible. Several BI companies made announcements about online services, features and new products. It's like "When Worlds Collide" - At the moment we're living in parallel worlds and neither world recognizes the other exists. Things will be shaken up in the BI market as Web tooling, BI infrastructure and data warehousing converge.

Mark Madsen is president of Third Nature, a consulting and research firm focused on business intelligence, data integration and data management. He is a principal author of Clickstream Data Warehousing and speaks about data warehousing and emerging technology. Write him at [email protected].Every TDWI conference leaves me with insight into what to expect during the next year. At last month's World Conference in San Diego, the main trends I saw were... predictive analytics... data governance... data warehouse databases and appliances... open-source BI... clickstream analysis... GIS/location intelligence... and the Web 2.0 dark horse...