HealthCare.gov: Lessons From Continuous Software Delivery

HealthCare.gov, President Obama's healthcare exchange website, has been heavily scrutinized this past year. In their quest to extend healthcare to every American, federal leaders put too many roadblocks and a tight deadline in the way of delivering a site that functioned and performed well during the initial rollout.

Like conventional manufacturing, software development is no easy feat, and the complexity is only continuing to increase. While the team of contractors who were hired to develop the site followed the principles of Agile, a rapid and lean software development process, it was not enough. In fact, neither the principles of Agile nor the succeeding practice of continuous integration could solve the software delivery challenge by themselves. What could help is operating in a continuous delivery model, which is described by many as the culmination of all of these development practices.

Continuous delivery is a software strategy that seeks to err on the side of delivering working software, as opposed to new features. The core idea is to create a repeatable, reliable, and iterative process by which working software is ushered from concept to customer. The foundation for this repeatable and reliable process usually means replacing manual, error-prone processes with automation.

[Users know what they want -- can you deliver? See Healthcare Software Development: We Can Do Better.]

The truth is that the journey to deliver a well-functioning, high-performance site is difficult, and the only way to keep up with increasing user demands is to evolve our software development processes to include these high levels of automation. There are two areas specifically that made HealthCare.gov a project difficult to pull off in the short time frame.

1. Too many control points
There are too many points of command and control across the application development lifecycle that do not offer visibility into who is doing what and when. The US government had about 55 contractors working on the site. They operated under a tight timeline and communicated with multiple agencies throughout the process. With critical stakeholders trying to make decisions under a timeframe that was less than suitable, it no doubt created major communication challenges. Teams need to be able to operate independently, and collaboratively, at a level that offers a continuous delivery of software updates.

2. Inadequate testing
Engineers who worked on fixing the HealthCare.gov site said there were stray lines of code that didn't seem to have a purpose. They found a lack of basic Web efficiency techniques and a failure to ensure the site could handle the website traffic load. If these issues were discovered and fixed ahead of the site moving into production, it could have saved the government a week's worth of distractions, a lot of embarrassment and political capital, and possibly tens of millions of taxpayers' dollars.

Automating as much of the testing process as possible and testing early and often are the keys to any software development project. Eliminating manual testing processes provides more immediate feedback to busy developers, allowing them to fix things rapidly. This accelerates cycle time and, as more features are added to the site, helps to verify the functionality of those changes and features more quickly. When activities like testing are not automated, the result is a labor-intensive process that can be time consuming and costly.

The HealthCare.gov site project underscores the complexity of building software today. While the project had many -- including the president -- scrambling for a fix, the extensive problems could have been addressed faster if a continuous delivery model was in place that combined Agile development, continuous integration, and DevOps. Operating in this model allows organizations to respond quicker to the end user to meet expectations. The larger the organization, the more important it will be to have this type of model in place.

According to an engineer on the project, the HealthCare.gov website contains about 500 million lines of software code. By comparison, a smartcar that connects to several systems has about 300 million lines of software code. Based on the number of lines of code in the HealthCare.gov project, there is no doubt that complexity of the system required good planning with clear, attainable goals from the start. Outlining those broader goals, as well as defining short-term milestones, is the key to any software development project today. The connected era that we live in requires real-time feedback and response, and the only way to get ahead is to measure progress and improve over time.

The HeathCare.gov debacle is arguably the most public software failure of 2013, and it sends a clear message that in today's demanding market software development must evolve into a modern approach. The movement to Agile was slow to move into the mainstream; over the past few years, organizations have embraced practices such as continuous integration to shorten cycle times. To compete in today's market, organizations need a fundamental approach that is a culmination of them all -- continuous delivery.

It doesn't matter whether your e-commerce D-Day is Black Friday, tax day, or some random Thursday when a post goes viral. Your websites need to be ready. Get the new Battle-Tested Websites issue of InformationWeek Tech Digest today. (Free registration required.)