Cerner Hosting Outage Raises Doctor, Hospital Concerns

Cerner Corp.'s remote hosting service went down for several hours on Monday, July 23, affecting Cerner's hospital and physician practice clients all over the country. This unusual outage, which reportedly took down the vendor's entire network, raises some new questions about the reliability of cloud-based hosting services.

Responding to an inquiry from InformationWeek Healthcare, Cerner spokesperson Kelli Christman said, "Cerner’s remote-hosted clients experienced unscheduled downtime this week. Our clients all have downtime procedures in place to ensure patient safety. The issue has been resolved and clients are back up and running. A human error caused the outage. As a result, we are reviewing our training protocol and documented work instructions for any improvements that can be made."

Christman did not respond to a question about how many Cerner clients were affected. But HIStalk, a health IT blog, reported that the outage was national and possibly international in scope.

[ Is it time to re-engineer your clinical decision support system? See 10 Innovative Clinical Decision Support Programs. ]

As for the duration of the downtime, HIStalk conveyed reader reports indicating that it had lasted all or a large part of the day. A typical comment: "We have not charted any patients today. Not acceptable from a healthcare leader." Colleen Becket, chairman and co-CEO of Vurbia Technologies, a cloud computing consultancy based in Tampa, Fla., told InformationWeek Healthcare that NCH Healthcare System, which includes two Tampa hospitals, had no access to its Cerner system for six hours. The outage affected both the facilities and NCH's ambulatory-care sites, she said.

"My husband, an ob/gyn physician [at NCH], called to tell me Cerner was down and could I please find out why," said Becket in an e-mail. "There had been no communication from hospital or Cerner with regards to downtime. They resorted back to handwritten orders during the six hours the system was down."

Another blogger on HIStalk said, "Communication was an issue during the downtime as Cerner’s support sites was [sic] down as well. Cerner unable to give an ETA on when systems would be back up. Some sites were given word of possible times, but other sites were left in the dark with no direction. Some sites only knew they were back up when staff started logging back into systems. Issue appears to have something to do with DNS entries being deleted across RHO network and possible Active Directory corruption."

Doug Hires, a partner at Santa Rosa Consulting, based in Bloomfield Hills, Mich., said the outage of the Cerner network was unusual and surprising. If one of the company's data centers had gone down, he noted, that wouldn't have been as big a problem because the data would have been backed up at another data center.

The federal HIPAA law, Hires explained, requires providers to have contingency procedures and backup plans in case their systems go down, so Cerner's clients were probably prepared for the outage. However, he added, it may have been difficult for providers to go back to writing orders on paper without having access to electronic health records containing patient medical histories.

Hires pointed out that service-level agreements (SLAs) with hosting companies include penalties for downtime. He surmised that if most Cerner customers had SLAs, this outage might prove to be very expensive for Cerner.

Whether a hospital or healthcare system operates onsite clinical and financial systems or outsources them to a remote hosting vendor, the results of downtime for end users would be the same. Nevertheless, Hires observed, the Cerner event shows "the potential and vulnerability is there" for an outage involving remotely served applications. This is a big concern for CIOs with regard to mission-critical information.

"The fear in the industry is about control," Hires noted. "If I have my own procedures and I have access to my own physical gear, I'm less concerned." For example, if an onsite server went down, a hospital might be able to swap it out with a redundant server that mirrored the data in real time.

A recent KLAS Research survey found that 55% of respondents already used remote hosting for clinical applications, storage, e-mail, and/or picture archiving and communication systems. Nearly a quarter of these respondents had remotely served electronic health records.

But another group of respondents were hesitant to have mission-critical applications served remotely, partly because of doubts about the reliability and security of cloud computing, KLAS reported.

In this InformationWeek Healthcare virtual event, EHRs: Beyond The Basics, experts will discuss how to improve electronic health record systems. It happens July 31.