Y2K, CEP, And The Mess On Wall Street

In December 1999, I remember trying to reassure a friend's brother -- who had just bought a pair of new shotguns from the local sporting goods store that he was planning to use for self-protection for himself and his teenage son -- that "the Y2K bug" was not likely to cripple the daily functioning of our society.As technical editor of a major software development magazine at the time, I thought I had some insight into the risks caused by the use of two-digit years in older mainframe software programs. I tried to assuage his fears by telling him that many or most of the nation's critical computer systems had been be tested for Y2K compliance and appropriately upgraded -- and, at any rate, many of these mainframe programs were on their last legs and not likely to be in daily use for mission-critical applications whose failure could cause widespread social disruption.

A newly divorced father, he was having none of it. I'm inclined to think he had been listening to some of the more strident doomsayers in the evangelical megachurch that he and his son belonged to who were trying to assign some apocalyptic significance to the year 2000, since a number of these worried church members also had been storing up money, boxes of food, and other supplies and getting ready to head for the hills in anticipation of some grand cataclysm. I'm reasonably sure that he or his church group had not been worried enough at the time to take all their money out of the bank ahead of time to ensure its safety, which many economists say was one of the catalysts for the Great Depression.

The news that the financial crisis that began 13 months ago has entered a new, far more serious phase, which The Wall Street Journal trumpets as "Worst Crisis Since '30s, With No End Yet in Sight," got me to thinking about the "Y2K crisis" by comparison.

I've spent a good part of the past week working on an overview of complex event processing (CEP), a technology used to do real-time analysis that was pioneered in the financial services, intelligence, and telecommunications industries and which is just now spreading to industries like manufacturing, casinos, and retailers, which are using CEP to build a new generation of real-time business intelligence apps that can analyze customer transactions to determine patterns of spending to better anticipate and influence spending. One of the analysts I talked with at length about CEP and the current financial crisis was Adam Honoré, a senior analyst at Aite Group who specializes in brokerage and financial services technology.

According to Honoré, "human factors such as poor risk management, not computerized trading," are primarily responsible for the market's current volatility. Wall Street firms have used trading algorithms to search patterns in data for buying or selling opportunities for decades but, Honoré says, the present instability is because of the risk posed by collateralized debt obligations and other unregulated asset-backed security and structured credit products.

Honoré thinks CEP technology can solve many of the risk management and compliance problems that financial firms face, and I tend to agree with him. CEP engines from vendors such Progress Apama, StreamBase, Aleri, Coral8, Tibco, IBM (which recently acquired CEP vendor AptSoft), and Oracle (through its purchase of BEA) have the potential to remove much of the risk in the market because they can absorb a wide range of information and normalize, aggregate, and analyze it in real time. This will provide greater transparency and visibility to market information, not only for regulatory agencies but also for enterprises that want to better manage risk and stop regulatory infractions before the regulator spots them.

I'm inclined to think that with hindsight the current financial crisis will look closer to the Y2K crisis than the Great Depression on a national calamity crisis scale, but I also think it wouldn't be a bad idea to hedge our bets by further exploring a technological solution like CEP to help regulate and take the risk out of chaotic financial markets.