Speech Recognition Tool Comes Up 'Speechless'

Health IT Boosts Patient Care, Safety

Breast-imaging reports prepared using a speech-recognition system are eight times more likely than conventional dictation transcription reports to contain major errors, according to a study of 615 such documents.

The study, which is published in the October issue of the American Journal of Roentgenology, relied on reports from the University Health Network in Toronto, Canada.

The breast-imaging reports, which were reviewed from January 2009 to April 2010, were almost evenly divided into two categories. In one, 307 reports used conventional dictation transcription in which the radiologist dictates the report and a team transcribes and reviews the report. The other 308 reports used automatic speech recognition (ASR) in which the radiologist dictates the report and software immediately transcribes the report onto a computer screen. The voice-recognition software used was Nuance's SpeechMagic (version 6.1, service pack 2). Dictation was conducted using a handheld speech microphone, the Pro-Plus LFH5276 from Philips Healthcare.

Anabel Scaranelo, lead author of the report, said the study's findings suggest that breast radiologists need more time for proof reading and review, and noted that they should probably have a 6- to 12-hour timeframe to look over the notes rather than sending the notes as soon as they are dictated using the speech-recognition software.

"Basically the study's findings tell us that despite the speed and other benefits that speech-recognition technology provides, it has its flaws, too, and we need to recognize them and find creative solutions to minimize their impact," Scaranelo told InformationWeek Healthcare. "There is pressure on radiologists to produce breast-imaging reports in the earliest time possible. However, the majority of [exams] are non-urgent exams. In that context, reports for breast imaging need a longer turnaround time to be proofread and edited."

Researchers also said that at least one major error was found in 23% of ASR reports, as opposed to 4% of conventional dictation transcription reports. Additionally, the error rate was even higher in breast MRI reports (35% of ASR compared with 7% of conventional reports). The lowest error rates occurred in reports of interventional procedures (13% of ASR and 4% of conventional reports) and mammography reports (15% of ASR and no conventional reports).

Major errors were those considered to affect the understanding of the breast-imaging report’s findings or errors affecting patient care. Errors that had no effect on the report's understanding or patient care were labeled minor errors.

"Errors in patients' reports [as the study's researchers point out], are not a simple by-product of speech software (or any single tool); it is indicative of a broken process where editing and review is not happening as it should," Joe Petro, senior vice president of engineering and R&D for Nuance Healthcare, said in an interview.

According to Petro, medical speech recognition is one of many technologies that clinicians use as part of the care delivery process; it is considered a "supervised tool" that is specifically used for increasing physician productivity. Petro also said supervision is key because the review, modify, and edit part of the workflow is a critical quality-assurance measure that helps to ensure accurate and high-quality clinical documentation.

"Speech recognition accuracy has never promised 100% (unsupervised) accuracy and for this reason 'supervision' is an institutionalized best practice leveraged by hundreds of thousands of physicians nationwide that use speech to produce documentation. If the 'supervision' piece of the workflow is missing, inadequate, or not effective, the result can be poor-quality documentation," Petro said.

He suggested institutions invest in training and measurement systems as part of speech technology rollouts to verify that these best practices are put in place.

"In the end, the use of speech is a paradigm shift for an organization and a physician and it requires change management. All practitioners know that the care team is accountable for making sure patients' medical reports are comprehensive and accurate," Petro added.

Other findings of the report were that major errors affecting patient care "were often caused by an incorrect unit of measure (millimeter/centimeter) or a missing or added 'no,' such as 'mammographic signs of malignancy' instead of 'no mammographic signs of malignancy.'"

Researchers defined 10 other types of errors, including word omission, word substitution, nonsense phrase, wrong word, punctuation error, added word, incorrect verb tense, plural error, spelling mistake, incomplete phrase.

Researchers also took into account variations in accents, and the level of experience of those using the speech recognition system. However, they found that these factors did not affect error rate as much as the method of generation. The report said: "The error rates did not differ substantially between reports generated by staff radiologists and trainees or between reports generated by speakers who spoke English as their first language and those whose native language was not English. After adjustment for academic rank, native language, and imaging modality, reports generated with ASR were eight times as likely as conventional dictation transcription reports to contain major errors."

Despite its problems, Scaranelo said the technology still has its place and is a useful tool for clinicians. "I would say to the radiologists: keep using speech-recognition technology, but conduct a more thorough process of proof reading, editing, and verifying the information to mitigate the errors."

Find out how health IT leaders are dealing with the industry's pain points, from allowing unfettered patient data access to sharing electronic records. Also in the new, all-digital issue of InformationWeek Healthcare: There needs to be better e-communication between technologists and clinicians. Download the issue now. (Free registration required.)