Speech Recognition Tool Comes Up 'Speechless' - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Healthcare // Clinical Information Systems
12:56 PM

Speech Recognition Tool Comes Up 'Speechless'

Breast imaging reports that use speech recognition are more likely to have errors than those generated by a transcriptionist.

Health IT Boosts Patient Care, Safety
(click image for larger view)
Slideshow: Health IT Boosts Patient Care, Safety
Breast-imaging reports prepared using a speech-recognition system are eight times more likely than conventional dictation transcription reports to contain major errors, according to a study of 615 such documents.

The study, which is published in the October issue of the American Journal of Roentgenology, relied on reports from the University Health Network in Toronto, Canada.

The breast-imaging reports, which were reviewed from January 2009 to April 2010, were almost evenly divided into two categories. In one, 307 reports used conventional dictation transcription in which the radiologist dictates the report and a team transcribes and reviews the report. The other 308 reports used automatic speech recognition (ASR) in which the radiologist dictates the report and software immediately transcribes the report onto a computer screen. The voice-recognition software used was Nuance's SpeechMagic (version 6.1, service pack 2). Dictation was conducted using a handheld speech microphone, the Pro-Plus LFH5276 from Philips Healthcare.

Anabel Scaranelo, lead author of the report, said the study's findings suggest that breast radiologists need more time for proof reading and review, and noted that they should probably have a 6- to 12-hour timeframe to look over the notes rather than sending the notes as soon as they are dictated using the speech-recognition software.

"Basically the study's findings tell us that despite the speed and other benefits that speech-recognition technology provides, it has its flaws, too, and we need to recognize them and find creative solutions to minimize their impact," Scaranelo told InformationWeek Healthcare. "There is pressure on radiologists to produce breast-imaging reports in the earliest time possible. However, the majority of [exams] are non-urgent exams. In that context, reports for breast imaging need a longer turnaround time to be proofread and edited."

Researchers also said that at least one major error was found in 23% of ASR reports, as opposed to 4% of conventional dictation transcription reports. Additionally, the error rate was even higher in breast MRI reports (35% of ASR compared with 7% of conventional reports). The lowest error rates occurred in reports of interventional procedures (13% of ASR and 4% of conventional reports) and mammography reports (15% of ASR and no conventional reports).

Major errors were those considered to affect the understanding of the breast-imaging report’s findings or errors affecting patient care. Errors that had no effect on the report's understanding or patient care were labeled minor errors.

"Errors in patients' reports [as the study's researchers point out], are not a simple by-product of speech software (or any single tool); it is indicative of a broken process where editing and review is not happening as it should," Joe Petro, senior vice president of engineering and R&D for Nuance Healthcare, said in an interview.

According to Petro, medical speech recognition is one of many technologies that clinicians use as part of the care delivery process; it is considered a "supervised tool" that is specifically used for increasing physician productivity. Petro also said supervision is key because the review, modify, and edit part of the workflow is a critical quality-assurance measure that helps to ensure accurate and high-quality clinical documentation.

"Speech recognition accuracy has never promised 100% (unsupervised) accuracy and for this reason 'supervision' is an institutionalized best practice leveraged by hundreds of thousands of physicians nationwide that use speech to produce documentation. If the 'supervision' piece of the workflow is missing, inadequate, or not effective, the result can be poor-quality documentation," Petro said.

He suggested institutions invest in training and measurement systems as part of speech technology rollouts to verify that these best practices are put in place.

"In the end, the use of speech is a paradigm shift for an organization and a physician and it requires change management. All practitioners know that the care team is accountable for making sure patients' medical reports are comprehensive and accurate," Petro added.

Other findings of the report were that major errors affecting patient care "were often caused by an incorrect unit of measure (millimeter/centimeter) or a missing or added 'no,' such as 'mammographic signs of malignancy' instead of 'no mammographic signs of malignancy.'"

Researchers defined 10 other types of errors, including word omission, word substitution, nonsense phrase, wrong word, punctuation error, added word, incorrect verb tense, plural error, spelling mistake, incomplete phrase.

Researchers also took into account variations in accents, and the level of experience of those using the speech recognition system. However, they found that these factors did not affect error rate as much as the method of generation. The report said: "The error rates did not differ substantially between reports generated by staff radiologists and trainees or between reports generated by speakers who spoke English as their first language and those whose native language was not English. After adjustment for academic rank, native language, and imaging modality, reports generated with ASR were eight times as likely as conventional dictation transcription reports to contain major errors."

Despite its problems, Scaranelo said the technology still has its place and is a useful tool for clinicians. "I would say to the radiologists: keep using speech-recognition technology, but conduct a more thorough process of proof reading, editing, and verifying the information to mitigate the errors."

Find out how health IT leaders are dealing with the industry's pain points, from allowing unfettered patient data access to sharing electronic records. Also in the new, all-digital issue of InformationWeek Healthcare: There needs to be better e-communication between technologists and clinicians. Download the issue now. (Free registration required.)

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Dr. Z
Dr. Z,
User Rank: Apprentice
9/30/2011 | 5:35:32 PM
re: Speech Recognition Tool Comes Up 'Speechless'
I have been involved in the integration of speech recognition technology with electronic health records for several years now and am a practicing physician in a busy Emergency Department. The Dragon software has gotten more accurate with every new release. In our experience the major problems with errors are not related to the software but with inadequate training, user mechanics and microphone quality. Most installs I've been associated with have used the Nuance handheld PowerMic II with Dragon Medical software v10.0 or newer. This combination has provided great speed & accuracy. In my opinion the poor results of the study are not software related, but are most likely caused by a combination of poor user mechanics, poor training and inferior microphone quality.
Brian Zimmerman, MD
Epic Physician Champion
Premier Health Partners
Co-founder of Physician Technology Partners
User Rank: Apprentice
9/30/2011 | 12:45:41 PM
re: Speech Recognition Tool Comes Up 'Speechless'
I find these results surprising. We did a formal study a few years ago comparing Nuance's Dragon Naturally Speaking Medical with transcription for visit notes and found both to have equally low error rates with no major errors.
Larry Garber, MD
Medical Director for Informatics
Reliant Medical Group (formerly known as Fallon Clinic)
Lisa Henderson
Lisa Henderson,
User Rank: Apprentice
9/29/2011 | 8:47:07 PM
re: Speech Recognition Tool Comes Up 'Speechless'
I really don't find this surprising. It doesn't seem there is a good voice recognition or voice response system out there. But in regard to this statistic: "Researchers also said that at least one major error was found in 23% of ASR reports, as opposed to 4% of conventional dictation transcription reports," that is just cause for needless alarm among patients and practitioners.

Lisa Henderson, InformationWeek Healthcare, contributing editor
Think Like a Chief Innovation Officer and Get Work Done
Joao-Pierre S. Ruth, Senior Writer,  10/13/2020
10 Trends Accelerating Edge Computing
Cynthia Harvey, Freelance Journalist, InformationWeek,  10/8/2020
Northwestern Mutual CIO: Riding Out the Pandemic
Jessica Davis, Senior Editor, Enterprise Apps,  10/7/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
[Special Report] Edge Computing: An IT Platform for the New Enterprise
Edge computing is poised to make a major splash within the next generation of corporate IT architectures. Here's what you need to know!
Flash Poll