Healthcare.gov: Hard-Earned Lessons For CIOs

Where 2016 US Presidential Contenders Stand On Tech Issues

Imagine the following IT project challenge -- create a system scalable for tens of thousands of concurrent active users with 99.9% uptime that touches and integrates with at least 20 other existing massive systems that are not under your control. You must execute delivery of this system within a culture where hundreds, maybe even thousands, of stakeholders provide ongoing input and not one of them has ultimate decision-making power. Because of this culture, the requirements for the system keep changing as you build it.

You must also deliver this complete system by an unchangeable date, and everyone is watching for you to fail.

This was the challenge faced by Henry Chao, the technology lead for the initial Healthcare.gov rollout, and his team in creating Healthcare.gov. Healthcare.gov is quietly humming along today, having enrolled 6 million people. But back then, Chao was tasked with and led the effort to create a massive IT system to implement the Affordable Care Act, a cornerstone of the Obama presidency intended to deliver health insurance and affordable healthcare to millions who otherwise couldn't get it. The project was already a politically controversial before it ever crossed Chao's desk.

[What are some of the most interesting and inspiring IT projects in the last year? Get an idea. Read 2016 InformationWeek Elite 100 Finalists Revealed.]

It came during a time when the US government was initiating the positions of federal CIO and federal CTO and pushing a cloud computing and digital agenda for the government. This made the Healthcare.gov project a marquee for what digital government could be.

And the directive for the Healthcare.gov site came with stipulations, such as one from the White House: "The experience shall be a world-class experience."

"How do you meet that metric? What constitutes a world-class experience?" Chao said. "When too many people throw those types of requirements in, the actual target becomes unclear."

Figure 1:

Healthcare.gov is more than just a website.

(Image via Healthcare.gov)

It's been a few years since Healthcare.gov was created, made its debut, and was widely and publicly criticized for not delivering out of the gate. Chao has since retired from his role at the head of the project and now serves as a private consultant. But InformationWeek caught up with Chao recently to discuss lessons learned, from his perspective, from this massive marquee project for a new age of digital government, what went wrong, and what went right.

It's Not Just a Website

The first thing you need to know about Healthcare.gov is that the website is just the front door. The backend challenge was one that could give even the best IT leaders a nightmare headache that could last for years.

"Healthcare.gov is not just a website," Chao told InformationWeek. "That's really an oversimplification. Healthcare.gov is a complex eligibility verification and determination application that is integrated with a data services hub that serves as a broker for all the requesters and responses that are coming from those authoritative sources."

That's a mouthful. Imagine being the person in charge of designing and implementing it.

Just the eligibility and enrollment components alone of the massive system touch up to 20 other systems. Consumers coming to Healthcare.gov to verify eligibility and to enroll in the program would need to have their identity verified, their income verified, and their social security numbers securely input. These pieces alone touch multiple external systems. Consumers may get stopped in the middle of the process if they need to provide some documentation to, say, verify income, that they didn't have handy.

This year's open enrollment period ended Jan. 31. Of the 12.7 million consumers enrolling in marketplace coverage this year, over 9.6 million came through the HealthCare.gov website. The system is now quietly humming along, even as its initial failure continues to garner headlines and scrutiny.

Launch Day Challenges

But in spite of the giant effort by hundreds of IT workers, contractors, vendors, and others, the closely watched initial roll out of Healthcare.gov did not go smoothly when it happened in Oct. 1, 2013. In the frequently recounted story, on the day of launch, the system was hit with 20x to 30x the expected traffic. The traffic wasn't just coming from people who were trying to enroll. It was coming from places all over the world, including China, Russia, and South America, according to Google Analytics.

"There were so many people interested in the rollout," Chao said about the traffic. "And a fraction of the traffic tried to create accounts, and they got stuck in the authentication process" due to the overload. The bottleneck eased in a week and a half, but then the number of people who were trying to use the system concurrently overwhelmed it.

A Solid Foundation, Plus...

Nonetheless, the system was built on a solid foundation by the hundreds of workers, contractors, and vendors who developed it initially. After the go-live challenges, a team of tech workers from Silicon Valley was called in to improve the load balancing, and to deploy additional hardware and software instances and virtual machines. The team was also meant to increase the number of concurrent active users that the system could handle. The system was handling all those users well by 70 days after launch. That's not something that could have happened if the foundation of the system had not been a solid one, Chao said.

"I think it's really a shame that there is so much to be learned here, but it often gets paved over with these glitzy stories about the small team of smart people from [Silicon Valley] that fixed this problem," he said. "It does a disservice to the hundreds of contractors, vendors, and career people who worked on this and gave their lives to this for four years and not just for 70 days."

NoSQL Provides Flexibility for the Project's Changing Requirements

Among the technology decisions that has led to long-term stability and success for the project was the choice of a noSQL database backend instead of a more traditional relational backend, according to Chao. The selection of the vendor MarkLogic provided the flexibility that was needed to develop a system that was changing even as it was created.

When Chao was creating this system, it was still unclear which US states would build their own exchanges or marketplaces. He knew he would have to integrate with the Medicaid system, even though it wasn't going to put up its own exchange. Chao ticked off other considerations -- premium tax credits, household

[Page 2: An Agile Approach]

Rising stars wanted. Are you an IT professional under age 30 who's making a major contribution to the field? Do you know someone who fits that description? Submit your entry now for InformationWeek's Pearl Award. Full details and a submission form can be found here.

composition aspects, authoritative sources of data, IRS information, Social Security information, and lawful presence information. The law stipulated that all these upstream systems and authoritative sources needed to be integrated into all these downstream processes.

"With relational models you have to have a very good logical physical model to work with, which means you have to have a lot of the requirements known up front. Otherwise you are constantly refactoring. NoSQL, MarkLogic in this case, allows you to adapt to those changes over time and be able to adjust how you are going to capture, process, store, and disseminate that data downstream to other processing entities, as well as into a warehouse environment."

An Agile Approach for Changing Requirements

All of that may very well make a noSQL approach better suited to Agile development, Chao said.

"You have this massively complex endeavor with those constraints, and you have to deliver it by Oct. 1. You need to choose technologies such as Mark Logic that are highly adaptive to a dynamic changing set of requirements, so what you need to focus on when looking at the requirements are the core of what [former US CIO] Steve VanRoekel called the minimum shippable product. It isn't about having this comfortable number of requirements, and everything is sequenced the way you need it, and the resources are at your fingertips. It's about working with what you have and never losing site of delivering on day 1. That's the most important thing."

Figure 2:

(Image: Henry Chao via LinkedIn)

Chao pointed to Cover Oregon, that state's own attempt at a website to implement the Affordable Care Act. The site did not deploy on time and was still unable to enroll people by April 2014, when its board opted to scrap the work on the project and go with the federal website instead. The state of Oregon is still embroiled in a legal battle with relational database vendor Oracle over the system.

Massive, Complex Systems

Have there been other IT systems deployed that operate with the same level of backend complexity while delivering the end-user a relatively simple interface? There are, but they did not face the same delivery time frame or optimization time frame. Chao points to the airline industry reservation system that was created in the 1960s.

[The US Department of Defense is boosting security with Windows 10.]

It consolidated all reservation systems and enabled consumers to buy tickets and have their price and seat guaranteed, regardless of whether they purchased from a travel agent or the airline itself. This is the same system that has since evolved in modern times to enable consumers to buy tickets online, via ticket agents, or directly from the airline.

"When a person purchases a ticket, you know that seat is available on that date and on that airline for that flight at that price," Chao said. "Building Healthcare.gov isn't just about the experience of the user. It's about an entire infrastructure that is similar to a centralized booking system that is looking at all those b2b or g2g transactions in order to be able to complete that lengthy business process of enrollment into a healthcare product."

Lessons Learned

A few years down the road, with benefit of hindsight, what are the lessons learned from the Healthcare.gov IT experience? Chao, now a consultant in the private sector, offered some perspective. Unsurprisingly, it's not about the right or wrong technology choices.

You need a single accountable person who is going to speak for how the system gets delivered, according to Chao. This person will indeed have committees of people offering input, however: "At the end of the day, it's clear who is in charge and who is responsible. This executive would have the ability to tap into the right resources and has decision-making authority to drive all the necessary orchestration or re-orchestration when priorities change."

Healthcare.gov did not have that situation.

Rising stars wanted. Are you an IT professional under age 30 who's making a major contribution to the field? Do you know someone who fits that description? Submit your entry now for InformationWeek's Pearl Award. Full details and a submission form can be found here.