Healthcare // Electronic Health Records
News
7/26/2012
02:23 PM
50%
50%

Cerner Hosting Outage Raises Doctor, Hospital Concerns

Cerner's national network went down for a day, forcing physicians to resort to handwritten orders.

11 Healthcare-Focused Business Intelligence Tools
11 Healthcare-Focused Business Intelligence Tools
(click image for larger view and for slideshow)
Cerner Corp.'s remote hosting service went down for several hours on Monday, July 23, affecting Cerner's hospital and physician practice clients all over the country. This unusual outage, which reportedly took down the vendor's entire network, raises some new questions about the reliability of cloud-based hosting services.

Responding to an inquiry from InformationWeek Healthcare, Cerner spokesperson Kelli Christman said, "Cerner’s remote-hosted clients experienced unscheduled downtime this week. Our clients all have downtime procedures in place to ensure patient safety. The issue has been resolved and clients are back up and running. A human error caused the outage. As a result, we are reviewing our training protocol and documented work instructions for any improvements that can be made."

Christman did not respond to a question about how many Cerner clients were affected. But HIStalk, a health IT blog, reported that the outage was national and possibly international in scope.

[ Is it time to re-engineer your clinical decision support system? See 10 Innovative Clinical Decision Support Programs. ]

As for the duration of the downtime, HIStalk conveyed reader reports indicating that it had lasted all or a large part of the day. A typical comment: "We have not charted any patients today. Not acceptable from a healthcare leader." Colleen Becket, chairman and co-CEO of Vurbia Technologies, a cloud computing consultancy based in Tampa, Fla., told InformationWeek Healthcare that NCH Healthcare System, which includes two Tampa hospitals, had no access to its Cerner system for six hours. The outage affected both the facilities and NCH's ambulatory-care sites, she said.

"My husband, an ob/gyn physician [at NCH], called to tell me Cerner was down and could I please find out why," said Becket in an e-mail. "There had been no communication from hospital or Cerner with regards to downtime. They resorted back to handwritten orders during the six hours the system was down."

Another blogger on HIStalk said, "Communication was an issue during the downtime as Cerner’s support sites was [sic] down as well. Cerner unable to give an ETA on when systems would be back up. Some sites were given word of possible times, but other sites were left in the dark with no direction. Some sites only knew they were back up when staff started logging back into systems. Issue appears to have something to do with DNS entries being deleted across RHO network and possible Active Directory corruption."

Doug Hires, a partner at Santa Rosa Consulting, based in Bloomfield Hills, Mich., said the outage of the Cerner network was unusual and surprising. If one of the company's data centers had gone down, he noted, that wouldn't have been as big a problem because the data would have been backed up at another data center.

The federal HIPAA law, Hires explained, requires providers to have contingency procedures and backup plans in case their systems go down, so Cerner's clients were probably prepared for the outage. However, he added, it may have been difficult for providers to go back to writing orders on paper without having access to electronic health records containing patient medical histories.

Hires pointed out that service-level agreements (SLAs) with hosting companies include penalties for downtime. He surmised that if most Cerner customers had SLAs, this outage might prove to be very expensive for Cerner.

Whether a hospital or healthcare system operates onsite clinical and financial systems or outsources them to a remote hosting vendor, the results of downtime for end users would be the same. Nevertheless, Hires observed, the Cerner event shows "the potential and vulnerability is there" for an outage involving remotely served applications. This is a big concern for CIOs with regard to mission-critical information.

"The fear in the industry is about control," Hires noted. "If I have my own procedures and I have access to my own physical gear, I'm less concerned." For example, if an onsite server went down, a hospital might be able to swap it out with a redundant server that mirrored the data in real time.

A recent KLAS Research survey found that 55% of respondents already used remote hosting for clinical applications, storage, e-mail, and/or picture archiving and communication systems. Nearly a quarter of these respondents had remotely served electronic health records.

But another group of respondents were hesitant to have mission-critical applications served remotely, partly because of doubts about the reliability and security of cloud computing, KLAS reported.

In this InformationWeek Healthcare virtual event, EHRs: Beyond The Basics, experts will discuss how to improve electronic health record systems. It happens July 31.

Comment  | 
Print  | 
More Insights
Comments
Threaded  |  Newest First  |  Oldest First
Cmonman
50%
50%
Cmonman,
User Rank: Apprentice
7/28/2012 | 1:56:52 PM
re: Cerner Hosting Outage Raises Doctor, Hospital Concerns
This calls for an investigation to determine the number of deaths, injuries, and near misses amidst the mayhem at each hospital that was affected. Cerner should publish the list of hospitals affected for the FDA, CDC, and the Joint Commission. Are electronic medical records all that safe? Doubt it when millions of patients face delays and hospitals are not equipped to handle it.
pbug
50%
50%
pbug,
User Rank: Strategist
8/30/2012 | 3:52:37 AM
re: Cerner Hosting Outage Raises Doctor, Hospital Concerns
I've read a lot of service contracts, and I've noticed that unless you go over them very carefully - both for general content and legalese, you may well not notice that vendors of 'cloud' and data services of various types have all sorts of 'outs' clauses hidden away. These tend to be one way - if the client blows it, heavy penalties. If the vendor blows it, OOPS! So Sorry!

My point is simple; these contracts generally are written by and favor the vendor. If you want there to be good penalty clauses, YOU and your attorney need to make sure they are included. High on the list of issues invoking penalties are outages.

I bank I once worked for had a data service used by at least 20,000 people. The contract called for the vendor to add a second site with separate comms and full, instant switchover capability. They never got around to it, but no one went after the vendor. One day, during a blackout, things were going OK - the huge data center on backup power, when a security guard heard an alarm going off, and he JUST HAD TO SILENCE IT. So he went, found the big red button marked Don't Touch Except In Case Of Emergency, and hit it! The alarms went silent, as did every other piece of equipment on the site, as the backup power shut down. Took the vendor over a week to clean up the mess and get fully back into service.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Dec. 9, 2014
Apps will make or break the tablet as a work device, but don't shortchange critical factors related to hardware, security, peripherals, and integration.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of December 14, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.