Application Insight: The Unstructured Revolution

If I've been using the word "revolution" too much in my columns of late, forgive me. But we clearly are in revolutionary times, at least to judge by the growing fervor over new paradigms for software delivery and pricing (as in software as a service), or everything else (as in Web/ Office/ Enterprise 2.0)

I'm not much of a proponent of revolutions, at least when it comes to supporting the notion that what is incremental and additive is actually revolutionary. Nonetheless, let me now propose that a real revolution is at hand, one that will not only define a radically different way in which we look at the world of data and business process, but one that may also help explain what is really happening at the core of some of the quasi revolutions I have previously disparaged.

This real revolution is the ability to develop, display, manage and otherwise leverage unstructured data across the enterprise. Before you stifle a yawn, consider this: Using unstructured data is what search is all about, what wikis and blogs actually try to do, what social computing was always hyped to facilitate. And this unstructured data problem is what makes out-of-the-box business processes so ho-hum and collaborative computing so darned hard to make truly valuable.

The relational database revolution that started the modern era of computing 20 years ago, and spawned pretty much everything we thought was cool until the Web and the Internet came along, was really about working very well with highly structured data. That revolution still has a long way to go--witness the disproportionate growth of data warehouse size relative to intrinsic value--but the opportunity pales in comparison to the next revolution: unstructured data.

The unstructured world is all around us, in the piles of papers that litter our desks and in the patchwork quilt of document formats--from desktop documents to CAD drawings to material safety data sheets to video and audio--that exist outside the realm of the relational database. Then there's all that unstructured data proliferating on the Internet: from buy/sell data on thousands of marketplaces to the blogs and wikis and Web sites that traffic in terabytes of data, without a single index field to unify them. Indeed, every business seems to have more unstructured data than structured data: Pervasive doesn't even begin to describe the problem; ubiquitous is probably the better word.

I'm declaring this a revolution because I'm starting to see some interesting applications and services that seek to deal with unstructured data in new ways. A couple of cool companies does not a revolution make, but the fact that the traditional enterprise software world (sorry, Web 2.0-ers) is tackling these issues gives me hope that the outcome will be something more than the latest buzzword-compliant new new thing.

One of the companies making hay out of the unstructured world is New Momentum, a start-up that trolls the online world for data about excess electronics inventories--what's for sale and who's buying at what price--and then turns all that unstructured data into some very valuable solutions for managing risk, pricing inventories, finding constrained parts and unmasking counterfeiters.

Another unstructured revolutionary is Accruent, which takes one of the biggest, messiest pieces of unstructured data around, the real estate contract, and mines it for new opportunities, cost savings and better business relationships for lessor and lessee alike. If you've never seen one of these contracts, consider yourself lucky. If these contracts are part of your day-to-day business, consider yourself luckier still that software is emerging that will deal with the mess.

How far can this unstructured revolution go? A long way, precisely because of the twin challenges of so much data and so little understanding about how to optimize its use. That understanding is coming, and when it matures, watch out. Not only will desktops around the world be a lot cleaner, but I predict our concepts of business efficiency and ROI will change for the better as well. Just in time for the next revolution, whatever that might be.