Mark Douglas saw his share of large-scale IT projects during 31 years in the US Air Force, including projects that have failed. But few projects offered as many lessons as the Expeditionary Combat Support System (ECSS), a $5 billion project that collapsed like a house of cards when the delivery of the initial "pilot" failed in its first release.

Douglas, a retired colonel and now an executive at Array Information Systems, says it's easy to second-guess how much the leadership team should have known about the problems the ECSS project faced and, if so, ask why "an ultra-important IT program" wasn't on "leadership's radar."

When an IT project such as ECSS crashes, it's usually the result of multiple causes. In the case of ECSS, the "software was not ready for prime time, and the IT infrastructure and data were woeful." But once an enterprise IT program "falls on its butt," Douglas says, the question for leaders needs to be, "How do you get it back on its feet when non-delivery is not an option?"

"This is a two-part solution -- what do you say and what do you do," Douglas said in an interview with InformationWeek.

When it comes to the talk, "honesty rules. Don't say everything you know, but present enough information to satisfy the typical questions, and at the same time set realistic expectations for what comes next."

For the walk, get brutally honest about what caused the failure. Surgically cut out bad staff, bad policies, bad processes, bad subcontractors, [and the elements contributing to] bad performance metrics/reporting. If you keep your communications eyes and ears open, you will soon see... all the stuff you need to keep," he said.

"This is critical because with a major, custom IT program you don't have the time, or usually the money, to bring on a new A-team that can learn and fix the program within the timeframe needed."

There are also pitfalls to avoid in a recovery.

"First, and most important, do not rush to a solution without a no-kidding understanding of the real reasons why this ship sank," said Douglas, warning not to rely on what the project owner or implementer says to their leadership. Think "real causes -- like the CSI show."

Second, "establish the actual, factual recovery plan -- what does it really take to get this ship afloat? What fixes, what process, what resources, what cost, what timeline?" You can expect this to be a tough assignment, he told us, "because the organization, your investors, customers, and your public want to dictate how and when you get this done."

Third, "don't over-promise, don't cover up, and don't try to recover with insufficient resources, corporate priority, or stakeholder communications."

[For a comprehensive look at what went wrong with the government's health insurance exchange site, see Obamacare Tech Saga: Special Report.]

When a project clearly fails to go as expected, when should a project owner consider bringing in a tech rescue team, as the White House did with HealthCare.gov?

"On mega, custom-coded IT implementations that trip before the finish line, many organizations want to throw money at the problem in the form of outsider IT SWAT teams. Keep your money," Douglas advises, "unless your program team is literally incapable of fixing the problems."

Bringing in an outside team on a gigantic IT program that has been in development for a couple years may sound appealing, "but the reality is that it is extremely difficult, expensive, and time consuming to bring in new people to work on custom code." He noted that the HealthCare.gov reportedly has more than 500 million source lines of code. "That's immense considering that Facebook reportedly runs on around 20 million source lines of code."

Part of learning to recover from a failed project also involves learning to heed the early warning signs of impending failure. When it became clear that CGI Federal, the lead contractor on the HealthCare.gov exchange site, was not close to completing crucial work within weeks of the launch (according to publicly disclosed documents), Douglas wondered, "Where was the warning of impending catastrophe? Why wasn't the government leadership informed, all the way to the top? That's like the general not knowing that forces are not ready for battle right before they are deployed."

Douglas noted that, while all the media attention has focused on the front end of the new government healthcare exchange website, of greater concern for project leaders are more obscure issues, including the back-end security of information on the site.

"In my last military assignment in Pacific Air Forces, commanders were briefed that our network was withstanding up to 15,000 cyberattacks per day. Name a person you know who has not had their credit card hacked. Security matters, big time," Douglas warns.

A recurring question in trying to avoid future government IT fiascos is whether agencies have the necessary capabilities in house to manage large-scale IT programs. From Douglas's point of view, "the US government is not the leader in commercial IT, nor should it be. IT business systems are not 'inherently governmental' federal provisions," he said, adding that there's a reason "major, complex IT programs typically require system (or solution) integrators."

"It's about division of labor and span of control," similar to the reason "a basketball team of all guards, or all forwards, or all centers will not prove to be the best team. Nor will a team that is coached by the players. The complexity of IT drives the need for specialists. The specialists don't necessarily play well together. An effective coach brings the best out of the team of specialists for the overall goals...

"Industry has proven time and time again that a competitive acquisition strategy nearly always yields the best overall outcomes." He noted that in the case of the HealthCare.gov exchange, the government awarded a massive, $678 million, no-bid contract to a private contractor and its team already doing work at the Centers for Medicare and Medicaid Services, the agency responsible for the exchange.

"The politically correct team often is not the correct team from a standpoint of capability and performance. Those characteristics are best served by competition."

Moving email to the cloud has lowered IT costs and improved efficiency. Find out what federal agencies can learn from early adopters in The Great Email Migration report. (Free registration required.)