Why Do Big IT Projects Fail So Often?

Why didn't the project use an iterative delivery approach to hone requirements and interfaces early? Why not start with site pilots and betas months or even years before the Oct. 1 launch date? The project was underway for more than three years, yet nothing was made available until Oct. 1. And why didn't the government effort leverage public cloud capabilities (or approaches) to enable efficient scaling? And where was the horizontal scaling design within the application to enable easy addition of capacity for unexpected demand?

These techniques appear to have been fully missed in the website implementation. Furthermore, the website code appears to be sloppy, not even using common caching techniques to improve performance. So in addition to suffering from weak sponsorship and ambiguous requirements, this program failed to leverage well-known technology and design best practices.

One would have thought that given the scale and expenditure on this program, the government would have assigned top technical resources and applied those best practices. Now the feds are scrambling with a "surge" of tech resources for the site.

While I wish the new project leaders and implementers well, this surge will bring its own problems. Ideas introduced now may not be accepted or integrated easily. And if the project couldn't handle the "easy" technical work -- sound website design and horizontal scalability -- how will it handle the more difficult challenges of data quality and security?

What To Do?

Clear sponsorship and proper governance are table stakes for any big IT project, but in this case more radical changes are in order. Why have all 36 states and the federal government roll out their healthcare exchanges in one waterfall or big bang approach? The sites that are working reasonably well (such as the District of Columbia's) developed them independently. Divide the work up where possible, and move to an iterative or spiral methodology. Deliver early and often.

Perhaps even introduce competitive tension by having two contractors compete against each other for each such cycle. Pick the one that worked the best and then start over on the next cycle. But make them sprints, not marathons. Three- or six-month cycles should do it. The team that meets the requirements, on time, will have an opportunity to bid on the next cycle. Any contractor that doesn't clear the bar gets barred from the next round so that there's no payoff for a contractor encouraging endless changes. And you have broken up the work into more doable components that can then be improved in the next implementation.

Finally, use only proven technologies. And why not ask the CIOs or chief technology architects of a few large-scale Web companies to spend a few days reviewing the program and designs at appropriate points. It's the kind of industry-government partnership we would all like to see.

If you want to learn more about how to manage (and how not to manage) large IT programs, I recommend "Software Runaways," by Robert L. Glass, which documents some spectacular failures. Reading the book is like watching a traffic accident unfold: It's awful but you can't tear yourself away. I expand on the root causes of and remedies for IT project failures in my blog post on project management best practices. And how about some government IT projects that went well? Here's one site's top 10 for 2012.

What project management best practices would you add? Please weigh in with a comment below.