HealthCare.gov: Lessons From Continuous Software Delivery - InformationWeek
IoT
IoT
Healthcare // Policy & Regulation
Commentary
10/13/2014
11:30 AM
Anders Wallgren
Anders Wallgren
Commentary
Connect Directly
Twitter
RSS
50%
50%

HealthCare.gov: Lessons From Continuous Software Delivery

A tight cycle of development, design, and testing is the best way to keep a big project (even a supposedly Agile one) from blowing up.

HealthCare.gov, President Obama's healthcare exchange website, has been heavily scrutinized this past year. In their quest to extend healthcare to every American, federal leaders put too many roadblocks and a tight deadline in the way of delivering a site that functioned and performed well during the initial rollout.

Like conventional manufacturing, software development is no easy feat, and the complexity is only continuing to increase. While the team of contractors who were hired to develop the site followed the principles of Agile, a rapid and lean software development process, it was not enough. In fact, neither the principles of Agile nor the succeeding practice of continuous integration could solve the software delivery challenge by themselves. What could help is operating in a continuous delivery model, which is described by many as the culmination of all of these development practices.

Continuous delivery is a software strategy that seeks to err on the side of delivering working software, as opposed to new features. The core idea is to create a repeatable, reliable, and iterative process by which working software is ushered from concept to customer. The foundation for this repeatable and reliable process usually means replacing manual, error-prone processes with automation.

[Users know what they want -- can you deliver? See Healthcare Software Development: We Can Do Better.]

The truth is that the journey to deliver a well-functioning, high-performance site is difficult, and the only way to keep up with increasing user demands is to evolve our software development processes to include these high levels of automation. There are two areas specifically that made HealthCare.gov a project difficult to pull off in the short time frame.

1. Too many control points
There are too many points of command and control across the application development lifecycle that do not offer visibility into who is doing what and when. The US government had about 55 contractors working on the site. They operated under a tight timeline and communicated with multiple agencies throughout the process. With critical stakeholders trying to make decisions under a timeframe that was less than suitable, it no doubt created major communication challenges. Teams need to be able to operate independently, and collaboratively, at a level that offers a continuous delivery of software updates.

2. Inadequate testing
Engineers who worked on fixing the HealthCare.gov site said there were stray lines of code that didn't seem to have a purpose. They found a lack of basic Web efficiency techniques and a failure to ensure the site could handle the website traffic load. If these issues were discovered and fixed ahead of the site moving into production, it could have saved the government a week's worth of distractions, a lot of embarrassment and political capital, and possibly tens of millions of taxpayers' dollars.

Automating as much of the testing process as possible and testing early and often are the keys to any software development project. Eliminating manual testing processes provides more immediate feedback to busy developers, allowing them to fix things rapidly. This accelerates cycle time and, as more features are added to the site, helps to verify the functionality of those changes and features more quickly. When activities like testing are not automated, the result is a labor-intensive process that can be time consuming and costly.

The HealthCare.gov site project underscores the complexity of building software today. While the project had many -- including the president -- scrambling for a fix, the extensive problems could have been addressed faster if a continuous delivery model was in place that combined Agile development, continuous integration, and DevOps. Operating in this model allows organizations to respond quicker to the end user to meet expectations. The larger the organization, the more important it will be to have this type of model in place.

According to an engineer on the project, the HealthCare.gov website contains about 500 million lines of software code. By comparison, a smartcar that connects to several systems has about 300 million lines of software code. Based on the number of lines of code in the HealthCare.gov project, there is no doubt that complexity of the system required good planning with clear, attainable goals from the start. Outlining those broader goals, as well as defining short-term milestones, is the key to any software development project today. The connected era that we live in requires real-time feedback and response, and the only way to get ahead is to measure progress and improve over time.

The HeathCare.gov debacle is arguably the most public software failure of 2013, and it sends a clear message that in today's demanding market software development must evolve into a modern approach. The movement to Agile was slow to move into the mainstream; over the past few years, organizations have embraced practices such as continuous integration to shorten cycle times. To compete in today's market, organizations need a fundamental approach that is a culmination of them all -- continuous delivery.

It doesn't matter whether your e-commerce D-Day is Black Friday, tax day, or some random Thursday when a post goes viral. Your websites need to be ready. Get the new Battle-Tested Websites issue of InformationWeek Tech Digest today. (Free registration required.)

Anders Wallgren brings with him more than 15 years of in-depth experience designing and building commercial software. Prior to joining Electric Cloud, he held executive positions at Aceva, Archistra, and Impresse. He also held management positions at Macromedia, Common Ground ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
4johnny
50%
50%
4johnny,
User Rank: Apprentice
11/9/2014 | 9:39:31 PM
HealthCare.gov initiative was not "Agile"
I would challenge anyone to present evidence that the HealthCare.gov initiative was operating in even a basic Agile manner.  When you leave the core testing to the last 2-3 weeks before launch (I believe), you are not following even basic Agile tenets.

Agile itself emphasizes working software.  If nothing else, just read the Agile Manifesto, taking care not to overlook its Twelve Principles.  Here are three relevant ones:
  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  • Working software is the primary measure of progress.

Notwithstanding, being Agile is not easy, and the Manifesto does not implement itself.  I definitely agree that a Continuous Delivery (CD) approach (i.e., based on automation) is the best practical way of achieving agility, and would have helped mitigate a lot of the risk.  IMO, CD provides fundamental methodology and tools for the future of software engineering.  Because software is eating the world (per Marc Andreessen), those who practice CD will out-compete those who don't.
Gary_EL
50%
50%
Gary_EL,
User Rank: Ninja
10/14/2014 | 1:23:04 PM
Too many cooks spoil the broth
.... especially when the meal is assembled at the dining room table, and not in the kitchen where it should have been.
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
10/13/2014 | 4:22:41 PM
A DevOps approach would have been doomed
Healthcare.gov needed agile development, continuous delivery and continuous integration, operating in a more of a DevOps approach. Easy to say this. But with so many parties involved, just about impossible to do.
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
Digital Transformation Myths & Truths
Transformation is on every IT organization's to-do list, but effectively transforming IT means a major shift in technology as well as business models and culture. In this IT Trend Report, we examine some of the misconceptions of digital transformation and look at steps you can take to succeed technically and culturally.
Video
Slideshows
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll