DevOps Turns Kaiser Permanente from Cranky to Capable

When an organization's software development teams each experience nearly a million dollars in lost productivity every month, it's time to take a hard look at the underlying development processes. Such was the realization that Kaiser Permanente made a couple of years ago.

But when a health care provider that employees more than 200,000 people, including more than 50,000 nurses and 20,000 physicians, looks to transform some part of itself, it doesn't happen overnight.

"It's like the Titanic," Alice Raia, vice president of digital presence technologies for Kaiser, told a packed presentation at the DevOps Enterprise Summit conference held this week in San Francisco. "Turning it is a two- or three-year event."

That didn't discourage Kaiser from starting down a DevOps path 18 months ago, with an initial focus on the company's KP.org web site. The situation was dire enough that the company had to correct its course. It was releasing code every couple of weeks, but every release had troubles. Kaiser was doing big, risky batch releases, and scope was constantly a moving target. Often, releases were delayed out of fear that some feature or capability would never make it into the code once it was released.

And then the company deduced that unforeseen obstacles and problems were causing each of its 75-person development scrum teams to lose an average of $864,000 in productivity. That number, said Raia, got the CIO's attention.

"The amount of inefficiency, breaks, and handoffs was slowing down the process," said Raia, who was brought in two years ago to manage digital engagement with Kaiser members, along with the company's software development lifecycle methodology and, thus, its DevOps practice. "We had absolutely nowhere to go but up."

To add further urgency, Kaiser is facing new market pressures that are putting a premium on agility, an area that hasn't exactly been one of the company's strengths.

"Our competition is really changing," said Raia. "Google just hired doctors. Amazon is getting into big data in health care. That's what we have to be able to react to."

DevOps as manufacturing

But when you're expecting to fill 25 million prescriptions, accommodate 27 million emails arriving in physicians' inboxes, enable 50 million lab tests to be viewed, and support 300 million visits to KP.org (most from mobile devices), agility can be elusive.

For that reason, Raia and her team decided to bite off what they could chew. That meant leaving the large, complex bi-monthly releases related to KP.org alone, and also resisting the urge to address daily content releases. Instead, they opted to focus their initial transformation efforts on bi-weekly releases, which are less complicated and come at a more manageable pace.

The team branded the new DevOps practice the "factory of the future," seeing it as a metaphor for thinking of their development process as a sort of manufacturing operation. This operational factory would support multiple applications and platforms, and it would be built by rethinking four threads related to software development:

It didn't take long for Raia to figure out that the last item on that list would be the trickiest.

"The technology is easy. The platform is easy. The culture and the people is where you need to spend most of your time," she said. "This requires a change in behavior that's difficult for people who've been with the organization for 10 or 15 years."

The cultural roadblocks Raia and her team met were numerous. Some people felt Raia didn't really understand how KP.org worked. Some thought they could wait out the effort in the hopes that she would leave and the DevOps transformation would collapse. Some cited regulatory hurdles as too difficult to transform. Others simply questioned whether the approach could work.

Sharing successes and failures

Raia and her team persisted, promoting each of their successes on social media, and sharing their failures to promote transparency, as well. In time, the steady progress broke down much of the resistance and the effort gained steam. Kaiser now has 600 users of the DevOps pipeline, including 500 of its 2,000 developers, and they are generating 450 code commits a day. They have 94 active projects, have worked on five automated test frameworks, and have had more than 20 production code releases since April.

What's more, its benchmarking has shown that Kaiser is now responding to service requests and change requests twice as fast as industry averages. And while there's still work to be done on KP.org, Raia sees huge opportunities to realize similar impact in Kaiser's pharmacy, claims, member support and sales organizations, as well as in supporting the workforce.

"A big cranky healthcare company that's been in business for 75 years, and we were able to do this," said Raia. "We got past the brick wall, and we've got some critical mass going."

If the Titanic can do it, anybody can.