=Paper=
{{Paper
|id=Vol-1178/CLEF2012wn-CLEFeHealth-SuominenEt2012c
|storemode=property
|title=Clinical Speech to Text Evaluation Setting
|pdfUrl=https://ceur-ws.org/Vol-1178/CLEF2012wn-CLEFeHealth-SuominenEt2012c.pdf
|volume=Vol-1178
|dblpUrl=https://dblp.org/rec/conf/clef/SuominenBJDHKYS12
}}
==Clinical Speech to Text Evaluation Setting==
https://ceur-ws.org/Vol-1178/CLEF2012wn-CLEFeHealth-SuominenEt2012c.pdf
Clinical Speech to Text
Evaluation Setting
Hanna Suominen1, Jim Basilakis2, Maree Johnson3, Linda Dawson4, Leif Hanlen1,
Barbara Kelly5, Anthony Yeo2, Paula Sanchez3
1
NICTA, National ICT Australia and The Australian National University, Locked Bag 8001,
2601 Canberra, ACT, Australia
2
University of Western Sydney, Locked Bag 1797, 2751 Penrith, NSW, Australia
3
Centre for Applied Nursing Research (a joint facility of the South Western Sydney Local
Health District & the University of Western Sydney), Locked Bag 7103, 1871 Liverpool BC,
NSW, Australia
4
University of Wollongong, 2522 Wollongong, NSW, Australia
5
The University of Melbourne, 2010 Melbourne, VIC, Australia
hanna.suominen@nicta.com.au, J.Basilakis@uws.edu.au,
Maree.Johnson@sswahs.nsw.gov.au, lindad@uow.edu.au,
leif.hanlen@nicta.com.au, b.kelly@unimelb.edu.au,
anthony.yeo@uni.sydney.edu.au,
Paula.Sanchez@sswahs.nsw.gov.au
Abstract. Failures in information flow from clinical handover are the leading
cause of sentinel events in the USA and associated with nearly half of all ad-
verse events and over a tenth of preventable adverse events in Australia. Verbal
clinical handover provides a good picture of the background clinical history and
current state of clinical management of a group of patients cared for by a nurs-
ing team. However, all this valuable verbal information is lost after three con-
secutive shifts if no notes are taken during handover. When traditional note-
taking by hand occurs, less than a third of data is transferred correctly after
five shifts.
We propose using an automated approach of cascading speech-to-text con-
version, standardisation with respect to controlled thesauri, and structuring in
accordance with documentation standards. This transcribes verbal handover in-
formation into written drafts for subsequent clinical review, editing, and addi-
tion to electronic health records.
In this paper, we introduce the evaluation setting for this technology devel-
opment in a laboratory environment. It ranks a wide range of recording devices
used alone or in combination with headsets and lapel microphones based on cli-
nicians’ preferences and their accuracy in speech-to-text conversion. The sam-
ple consists of four student nurses and four experienced academics from diverse
clinical specialties and speaking styles. To simulate realistic nursing clinical
handovers, twenty handover scenarios have been scripted. The subsequent eval-
uation in a clinical environment will address speech-to-text conversion, stand-
ardisation, and structuring with the short-listed devices in six hospitals with the
sample of thirty authentic handover situations per hospital.
To compare recorder-microphone combinations across all participants, pro-
fessional-level recording devices are used to record each participant. The re-
cordings are then played using professional-level speakers across all recorder-
microphone combinations to achieve equivalency in voice input. Statistical ac-
curacy in speech-to-text conversion with noise experimentation is used to de-
termine the most accurate combination. Two speech-to-text systems are com-
pared against transcription by hand.
An eighteen-item pre-experimental survey addresses initial perceptions of us-
ing the proposed automated approach in clinical settings. This includes partici-
pants’ opinion on the improvement of clinical handover with the proposed au-
tomated approach, their understanding of the related technologies and perceived
problems with the clinical application. An eleven-item post-experimental sur-
vey examines device usability with reference to the specific experimental de-
vices. Each participant is asked to complete both surveys and participate in a
one-to-one interview. All participants are videoed using the recording devices
and accessing typical device functions to further examine human-device inter-
actions for usability assessment.
We are seeking additional partners to further develop and evaluate the ap-
proach and setting.
Keywords: Evaluation; Natural Language Processing; Nursing Informatics;
Speech Recognition Software
1 Introduction
Failures in information flow from clinical handover are the leading cause of sentinel
events in the USA and associated with nearly half of all adverse events and over a
tenth of preventable adverse events in Australia.1-3 Verbal clinical handover provides
a good picture of the background clinical history and current state of clinical mana-
gement of a group of patients cared for by a nursing team. However, all this valuable
verbal information is lost after three consecutive shifts if no notes are taken during
handover. When traditional note-taking by hand occurs, less than a third of data is
transferred correctly after five shifts.4-5
We propose using an automated approach of cascading speech-to-text conversion,
standardisation with respect to controlled thesauri, and structuring in accordance with
documentation standards. This transcribes verbal handover information into written
drafts for subsequent clinical review, editing, and addition to electronic health re-
cords. We have already demonstrated the suitability of the document structure scienti-
fically and practically by introducing a documentation template to be populated by
typing. After its initial pilot testing in six wards, implementation across four major
teaching hospitals in Australia is nearing completion.6-7
Fig. 1. Laboratory environment for evaluation
In this work-in-progress paper, we introduce the evaluation setting for this techno-
logy development in a laboratory environment (Figure 1). This setting aims to define
hardware to be used in a subsequent evaluation in a clinical environment. It ranks
hardware alternatives based on clinicians’ preferences and their accuracy in speech-
to-text conversion when using fixed software. The subsequent evaluation in a clinical
environment will address not only speech-to-text conversion but also the steps of
standardisation and structuring.
2 Materials and Methods
A wide range of recording devices are considered and compared. These include an
MP3 player, medium and high-end voice recorders, smart phones and tablet compu-
ters. The devices are used alone or in combination with medium and high-end head-
sets as well as omnidirectional and noise-cancelling lapel microphones. The sample
consists of four student nurses and four experienced academics from diverse clinical
specialties and speaking styles, including accents and voice qualities. To simulate
realistic nursing clinical handovers, twenty handover scenarios have been scripted.
Derived from existing clinical handover data, these fictitious and de-identified scena-
rios reflect the full range of possible handover situations including structured hando-
ver, unstructured handover, group presentation and individual presentation. Each
handover scenario includes the use of proper English, jargon terms, fragmented lan-
guage, atypical abbreviations and clinical terminology. In a second phase, the short-
listed 3–5 recording devices are tested in clinical practice with the sample of 180
authentic handover situations (i.e., thirty situations in six hospitals). We have chosen
this two-phase approach to minimize the evaluation bias caused by the burden of wea-
ring multiple devices in clinical practice when compared with the final goal of having
one device only.
3 Results and Discussion
Evaluation of accuracy: To enable systematic comparison of recorder-microphone
combinations across all participants, professional-level recording devices are used to
record each participant. The recordings are subsequently replayed using professional-
level speakers across all recorder-microphone combinations to achieve equivalency in
voice input. Statistical accuracy in speech-to-text conversion is used to determine the
most accurate combination. This use of pre-recorded sound files also enables syste-
matic manipulation and experimentation of a wide range of noise levels and types
(e.g., ambient, intrusive, continuous, intermittent, and other people in group presenta-
tion). Minimally two speech-to-text systems are compared against transcription
by hand.
Personalisation to clinical context: An eighteen-item pre-experimental survey ad-
dresses initial perceptions of using the proposed automated approach in clinical set-
tings, prior to the introduction of experimental recording devices. This includes parti-
cipants’ opinion on the improvement of clinical handover with the proposed auto-
mated approach, their understanding of the related technologies and perceived pro-
blems with the clinical application. In addition to assessing the perceived benefits and
problems of recording devices, an eleven-item post-experimental survey examines
device usability with reference to the specific experimental devices. Each participant
is asked to complete both surveys and participate in a one-to-one interview or focus
group discussion. Our survey templates are available at
http://bit.ly/JB0yHR.
4 Conclusion
We are seeking additional partners to further develop and evaluate the approach and
setting in order to gain understanding across specialties, jargons, genres,
and languages.
Acknowledgements
NICTA is funded by the Australian Government as represented by the Department of Broad-
band, Communications and the Digital Economy and the Australian Research Council through
the ICT Centre of Excellence program. We gratefully acknowledge collaboration in cross-
country comparisons with Riitta Danielsson-Ojala, Heljä Lundgrén-Laine, and Sanna Salanterä
(University of Turku, Department of Nursing Science, Turku, Finland).
References
1. Joint Commission on Accreditation of Healthcare Organisations: Health Care at the
Crossroads: Strategies for Improving the Medical Liability System and Preventing Patient
Injury, 2005, http://bit.ly/KJ8ylN.
2. Australian Commission on Safety and Quality in Health Care: Windows into Safety and
Quality in Health Care 2008, 2008, http://bit.ly/JwlxXz.
3. Australian Commission on Safety and Quality in Health Care: The OSSIE Guide to Clini-
cal Handover Improvement, 2009, http://bit.ly/JWZ5vb.
4. Pothier D, Monteiro P, Mooktiar M, Shaw A: Pilot study to show the loss of important da-
ta in nursing handover. Br J Nurs. 2005;14(20):1090-3.
5. Matic J, Davidson PM, Salamonson Y: Review: bringing patient safety to the forefront
through structured computerisation during clinical handover. J Clin Nurs. 2011;20(1-
2):184-9
6. Johnson M, Jefferies D, Nicholls D: Exploring the structure and content of nursing clinical
handovers. Int J Nurs Pract. 2012, in press.
7. Johnson M, Jefferies D, Nicholls D: Developing and testing a minimum data set for elec-
tronic handover in nursing. J Clin Nurs. 2012;21(3-4):3331-43.