8 Expensive IT Blunders

Rule #6: Murphy's Law Applies

Not everyone hates IT miscues. Tax cheats, for instance, are thrilled about them when the organization at the center of the screw-ups is the Internal Revenue Service. That was the case earlier this year when the IRS botched an upgrade to its fraud-detection software. "If anything could go wrong, it went wrong regarding this particular project," says Margaret Begg, assistant inspector general for audit with the Treasury Department.

The IRS planned to launch the system in January, in time for the 2006 tax season and one year after the original implementation date. In anticipation, the tax agency's IT staffers pulled the plug on the old system. The new version, which is being developed and implemented by Computer Sciences Corp. and is fed by a database that holds more information than any other IRS system except its master database, didn't work. Not good for a piece of software that's been on the drawing board since 2001. Now, the Treasury Department's Office of Inspector General estimates that the lack of a functioning anti-fraud system cost $318 million in fraudulent refunds that didn't get caught.

A four-month investigation by the House Ways and Means Committee found "incompetence at all levels," committee chairman Bill Thomas, R-Calif., wrote in an August letter to Treasury Secretary Henry Paulson. Among other things, IRS IT leaders erred when they classified the effort as a maintenance project rather than a major upgrade. That led to insufficient oversight and funding, Thomas says, "despite the critical importance of the system ... in protecting federal revenue." Not to mention that the system is one of 19 IRS assets the federal government has deemed part of the nation's critical infrastructure that must be protected from terrorist attacks.

In January, when the system was supposed to go live, users generated 534 problem tickets. One problem was that the electronic fraud-detection system project office wasn't communicating with the Criminal Investigation division of the IRS, the main user of the system. In fact, communication was breaking down all over the place. When one development team made a change, it didn't notify other development teams whose portions of the system were affected by the change. As a result, the IRS's System Acceptability Testing team would encounter another problem in another place. This would cause the SAT team to write another problem ticket, which would go back to the contractor, requiring another software fix, according to a report by Begg. In an interview, Begg says the IRS lacked the project management discipline to pull off an upgrade of such a critical application. "They lacked testing rigor, and they didn't have program management activities set up where there's a defined plan and that they could execute against," she says.

This, of course, isn't the first big IT snafu for the IRS. The agency has been trying to upgrade its overall computing infrastructure for the past eight years, at a cost of more than $8 billion. A Government Accountability Office review of that effort found multiple instances of cost overruns, waste, and inefficiency. Part of the problem is that IRS IT personnel are retiring in droves and qualified replacements are tough to find. "They've lacked the stability and consistency you need to implement certain activities," says Begg, adding that IT management turnover has been particularly high. The IRS is in the process of booting up the old system in time for the 2007 tax season, while plans for the Web-based system have been shelved. Brace yourselves.

Rule #7: Check The Shelf Life

In 2005, the Federal Bureau of Investigation killed off its problem-plagued Virtual Case File system, a custom-developed software application that was supposed to let agents search across multiple criminal databases in the hunt for perps. That capability has been a Holy Grail for the bureau as far back as the 1980s, when software company Inslaw alleged that the Department of Justice purloined Inslaw's Promis case management software for use by the FBI. Inslaw ultimately lost the case. The need for such a system took on new urgency after the 9/11 terrorist attacks.

The failure of the $170 million Virtual Case File effort, part of the FBI's Trilogy IT modernization program, likely won't result in any criminal charges, but in launching the program the FBI did break one unspoken law of IT planning: Don't build a long-term project on technology that will be outdated by the time the project is completed. Project planners must think about what the overall tech environment will look like, and what technology will be widely used, once an initiative is completed.

In planning the Virtual Case File in 2001, the FBI chose custom-developed software, although cheaper, more flexible off-the-shelf apps were coming into favor among many large organizations. "The pace of technological innovation has overtaken our original vision for VCF, and there are now existing products that did not exist when Trilogy began," the FBI conceded in a report last year on the demise of the project.

The VCF project also suffered from having nine program managers and five CIOs in its short, unhappy four-year life.

To its credit, the bureau appears to have learned its lesson. It's ditched the VCF program in favor of an information management system called Sentinel. Sentinel will be based on cutting-edge technology and should have a long shelf life: It will be Web-based, and a service-oriented architecture ensures that it will be able to connect to external law enforcement databases and other information sources that incorporate Web services standards.

Sentinel's first phase will provide FBI agents, analysts, and other personnel with a portal that will let them access the soon-to-be-replaced automated case-support system and, later, data in the new case management system. It also will include a case management "workbox" that will summarize a user's workload (the case files an agent or analyst is working on) and provide automatic indexing in case files according to person, place, or thing.

In March, the FBI awarded Lockheed Martin a $305 million contract to help with the implementation of Sentinel. Subcontractors include Accenture and Computer Sciences Corp. CSC's Dyncorp unit was one of the original contractors on the Department of Justice's efforts to implement Promis. Let's hope things go better this time around.