Best practice for clinical handover and its documentation recommends standardized, structured, and synchronous processes with patient involvement. Cascaded speech recognition (SR) and information extraction could support their compliance and release clinicians' time from writing documents to patient interaction and education. However, high requirements for processing correctness evoke methodological challenges. First, multiple people speak clinical jargon in the presence of background noise with limited possibilities for SR personalization. Second, errors multiply in cascading and hence, SR correctness needs to be carefully evaluated as meeting the requirements. This overview paper reports on how these issues were addressed in a shared task of the eHealth evaluation lab of the Conference and Labs of the Evaluation Forum in 2015. The task released 100 synthetic handover documents for training and another 100 documents for testing in both verbal and written formats. It attracted 48 team registrations, 21 email confirmations, and four method submissions by two teams. The submissions were compared against a leading commercial SR engine and simple majority baseline. Although this engine performed significantly better than any submission [i.e., 38.5 vs. 52.8 test error percentage of the best submission with the Wilcoxon signed-rank test value of 302.5 (p < 10-12)], the releases of data, tools, and evaluations contribute to the body of knowledge on the task difficulty and method suitability. Contributor Statement: HS, LH, and GJFJ designed the task and its evaluation methodology. HS developed the dataset and together with LH, led the task as a part of the CLEFeHealth2015 evaluation lab, chaired by LG and LK. HS drafted the paper and after this all authors revised and approved it.
Fluent information flow, defined as channels, contact, communication, or links to
pertinent people [
In order to support compliance with these processes, cascaded speech recognition
(SR) with information extraction (IE) has been studied in 2015 [
This novel application evokes fruitful challenges for method research and
development, and consequently, its first part (i.e., clinical SR) was chosen as the Task 1a of
the eHealth Evaluation Lab by the Conference and Labs of the Evaluation Forum
(CLEF) in 2015 [
The task aligns with the CLEFeHealth usage scenario of easing patients, their
nextof-kin, and other laypersons in understanding and accessing electronic health
(eHealth) information [
The hold-out method was used in performance evaluation of the task. Task materials consisted of a training set of 100 synthetic patient cases and an independent set of another 100 synthetic patient cases for testing. Given the training set, the task was to minimize the number of incorrectly recognized words on the test set (i.e., on the heldout set). Performance of the submitted methods and two baseline methods was compared statistically. 2.1
The dataset called NICTA Synthetic Nursing Handover Data was used in this task for
method development and training [
First, the first author of this paper (Adj/Prof in machine learning and communication for health computing) generated 100 synthetic patient profiles, using common user profile generation techniques [21]. With an aim for balance in patient types, she created profiles for 25 cardiovascular, 25 neurological, 25 renal, and 25 respiratory 3 Derived from OECD.StatsExtracts (http://stats.oecd.org/) for 2009 (i.e., the most recent year that has almost all data available) 4 http://www.nicta.com.au/nicta-synthetic-nursing-handoveropen-data-software-and-demonstrations/ patients of an imaginary medical ward for adults in Australia. These patient types were chosen because they represent the most common chronic diseases and national priority areas [22]. The reason for patient admission was always an acute condition, but some patients had also chronic diseases. Some patients were recently admitted to the ward, some had been there for some days already, and some were almost ready to be discharged after a shorter or longer inpatient period. Each profile was saved as a DOCX file and contained a stock photo from a royalty-free gallery, name, age, admission story, in-patient time, and familiarity to the handover nurses.
Second, the first author supervised a registered nurse (RN) in creating the written handover documents for these 100 profiles. The RN had over twelve years’ experience in clinical nursing. Australian English was her second language and she was originally from the Philippines. She was guided to imagine herself working in the medical ward and delivering verbal shift-change handovers to another nurse by the patient’s bedside as if she was talking. All handover information was to be given as a 100–300-word monologue, using normal wordings. The resulting realistic but fully imaginary handovers were saved as TXT files.
Third, the first author supervised the RN in creating the verbal handover
documents by reading the written handover documents out loud as the nurse giving the
handover. She was guided to record in a quiet office environment, try to speak as
naturally as possible, avoid sounding like reading text, and repeat the take until she
was satisfied with the outcome. The Olympus WS-760M digital recorder [purchased
for 269.00 AUD (191 €) in October 2011, weight of 51 g, dimensions of 98.5 mm ×
40.0 mm × 11.0 mm] and Olympus ME52W noise-canceling lapel-microphone
[purchased for 15.79 AUD (11 €) in October 2011, weight of 4 g (+ 11 g for an optional
cable and clip), dimensions of 35 mm (+ a 15 mm plug) × 13 mm × 13 mm] were
used, because they were previously shown to produce superior word correctness in SR
[
Fourth, the first author used Dragon Medical 11.0 for SR.8 This clinical engine was
chosen because it included an option for Australian English. It was first initialized
with not only this accent but also to the RN’s age of 22–54 years and recording of The
Final Odyssey [DOCX file of 3,893 words in writing and WMA file of 29 minutes 22
seconds as speech (4 minutes needed)] using the aforementioned recorder and
microphone. Also these training/personalization/adaption files were released. Six Dragon
vocabularies (i.e., general as the most generic clinical vocabulary, medical because of
the medical ward, nursing because of the nursing handover, cardiology because of the
cardiac patients, neurology because of the neurological patients, and pulmonary
disease because of the respiratory patients) were compared although the nursing
vocabu5 http://www.olympus.co.uk/site/en/archived_products/audio/
audio_recording_1/ws_760m/index.pdf
6
https://shop.olympus.eu/UK-en/microphones/olympus/me52w-minimono-microphone-p-239.htm
7 http://sourceforge.net/projects/audacity/
8
http://www.nuance.com/products/dragon-medical-practiceedition/index.htm
lary shown to produce the best results in SR [
The data release with the requirement to cite [
The training set was supplemented with an independent dataset for testing. This additional set of 100 synthetic patient cases was developed in 2015. Each case consists of (1) a patient profile, (2) a written handover document, (3) a verbal handover document, and (4) a speech-recognized document with respect to the nursing vocabulary. Its subset of documents (3) and (4) was released on the task page on 23 April 2015; the organizers did not release the profiles in order to avoid their contents to be used an a processing input, and they held the written handover documents out as a blind set for anyone but the first author and RN to ensure independent training and testing. The set was created the same way as the training set except that the profile photos were reused, software was updated to Audacity 2.1.3 for Mac with ffmpeg-mac-2.2.2,11 and only the nursing vocabulary was chosen for SR.
The data release was approved at NICTA and the RN was consented in writing. The licensing constraints are the same as before; however, we ask to cite this task overview for the data release. 2.3
The participants needed to submit their processing results by 1 May 2015 using the Easy Chair System of the lab.12 Submissions that developed the SR engine itself were evaluated separately from those that studied post-processing methods for the speechrecognized text. Also a separate submission category was assigned to solutions based on both SR and text post-processing.
Only fully automated methods were allowed, that is, human-in-the-loop means were not permitted. Each participant was allowed to submit up to two methods/parameterizations/compilations (referred to as a method from now on) to the first category and up to two methods to the second category. If addressing both these categories, the participant was asked to submit all possible combinations of these methods as their third category submission (i.e., up to 2 × 2 = 4 files). 9 http://creativecommons.org/licenses/by-nc-nd/4.0/ 10 http://creativecommons.org/licenses/by/4.0/ 11 https://www.ffmpeg.org/ 12 https://easychair.org/conferences/?conf=clefehealth2015resul with the professional license
In addition to the submission category, the submissions consisted of the following elements: team name and description; address of correspondence; author(s); at least three method keywords;13 method description (max 100 words per method); and processing outputs for each method on the 100 training and 100 test documents. 2.4
We challenged the participants to minimize the number of incorrectly recognized words on the independent test set. This correctness was evaluated on the entire test set using the primary measure of the percentage of incorrect words [aka the error rate percentage (E)] as defined by the Speech Recognition Scoring Toolkit (SCTK), 2.4.0 without punctuation as a differentiating feature.14 This measure sums up the percentages of substituted (S), deleted (D), and inserted (I) words (i.e., E = S + D + I) and consequently, the smaller the value of E, the better the performance. To illustrate these error types, speech-recognizing your word as you are had the substitution (your, you), insertion are, and deletion word.
As secondary measures, we reported the percentage of correctly detected words (C) on the entire test set together with the breakdown of E to D, I, and S. We also documented the raw numbers of correct (nC), substituted (nS), deleted (nD), and inserted words (nI). Notice that C + S + D = 100 and nC + nS + nD is the number of words in the reference standard.
To provide more details on performance differences across the individual handover documents, we also computed the error rate percentage ei in individual documents di ∈ {d1, d2, d3, ... , d100} of the test set. Then, we summarized these values through their minimum (min), maximum (max), mean, median, and standard deviation (SD).
Finally, instead of evaluating this generalization capability of the method to unseen data, we assessed the resubstitution performance on the training set; a method that does not perform well even on its training set is poor, but excellence on training set may indicate over-fit, leading to issues in the generalizability. 2.5
We used two baseline systems in the task, namely Dragon and Majority. The Dragon baseline was based on Dragon Medical 11.0 with the nursing vocabulary and initialization to the RN, recorder, and microphone. This commercial system included substantial but closed domain dictionaries, had a substantial license fee per person [purchased for 1,600.82 AUD (1,139 €) in January 2013], and was limited to the Microsoft Windows operating system. Notice that the 200 synthetic handover cases were not used to train the Dragon baseline. The Majority baseline assumed that the right number of words is detected (i.e., the number of test words originating from the refer13 preferably Medical Subject Headings (http://www.nlm.nih.gov/mesh/MBrowser.html) or Association for Computing Machinery classes (http://www.acm.org/about/class/ccs98-html) 14 http://www.itl.nist.gov/iad/mig/tools/ ence standard) and recognized each word as the most common training word (i.e., and) with correct capitalization (i.e., and for and, And for And, and so forth).
To supplement the usage guidelines of SCTK,15 we provided the participants some helpful tips (Appendix 1): we released an example script for removing punctuation and formatting text files; a formatted reference file and Dragon baseline for the training set; overall and document-specific evaluation results for this file pair; and commands to perform these evaluations and ensure the correct installation of SCTK. 2.6
Statistical differences between the error rate percentages of the two baselines and
participant submissions were evaluated using the Wilcoxon signed-rank test (W) [
After ranking the baselines and submissions based on their error rate percentage on the entire dataset for testing, W was computed for the paired comparisons from the best and second-best method to the second-worst and worst method. The resulting p value and the significance level of 0.05 was used to determine if the median performance of the higher-ranked method was significantly better than this value for the lower-ranked method. All statistical tests were computed using R 3.2.0.16 3
The task released in both verbal and written formats the total of 200 synthetic clinical documents that can be used for studies on nursing documentation and informatics. It attracted nearly 50 team registrations with about half of them confirming their participation through email. Two teams submitted two SR methods each. Although no method performed as well as the Dragon baseline, the task contributed to the body of knowledge on the task difficulty and method suitability. 3.1
The task released a training set of 100 documents (Fig. 1) on 27 October 2014; an independent test set of 100 documents on 23 April 2015; and reference standard for the test documents together with processing results of the Dragon baseline and subKen harris, bed three, 71 yrs old under Dr Gregor, came in with arrhythmia. He complained of chest pain this am and ECG was done and was reviewed by the team. He was given some 15 http://www1.icsi.berkeley.edu/Speech/docs/sctk
1.2/options.htm#option_r_name_0 16 http://www.r-project.org/ anginine and morphine for the pain. Still tachycardic and new meds have been ordered in the medchart. still for pulse checks for one full minute. Still awaiting echo this afternoon. His BP is just normal though he is scoring MEWS of 3 for the tachycardia. He is still for monitoring. Dragon baseline: Own now on bed 3 he is then Harry 70 is 71 years old under Dr Greco he came in with arrhythmia he complained of chest pain this morning in ECG was done and reviewed by the team he was given some and leaning in morphine for the pain in she is still tachycardic in new meds have been ordered in the bedtime is still 4 hours checks for one full minute are still waiting for echocardiogram this afternoon he is BP is just normal though he is scarring meals of 3 for the tachycardia larger otherwise he still for more new taurine missions later in 2015.17 Errors that the Dragon baseline made on training documents ten or more times included substituting years for yrs (n = 48), in with and (n = 22), one with 1 (n = 17), alos with obs (n = 12), and to with 2 (n = 12); deleting is (20), are (13), and and (11); and inserting and (210), is (136), in (106), she (71), are (58), all (45), arm (44), for (43), the (37), he (35), that (34), a (27), her (19), eats (15), on (15), also (14), am (12), does (11), bed (10), s (10), and to (10) [26].
The training set had 7,277 words and 1,304 (1,377) of them were unique without (with) capitalization as a differentiating feature. For the test set, these numbers were 6,818, 1,279, and 1,323, respectively. Although both sets followed the Zipf’s law [27] (Fig. 2), and were thereby typical language samples, their vocabularies shared only 645 (738) unique words without (with) capitalization as a differentiating feature (Fig. 3). Consequently, the sets can be seen as fairly independent, as intended. The 10 common words without capitalization as a differentiating feature were and (347 occurrences in the training asset and 315 in the test set), is (256, 288), he (243, 201), in (170, 212), for (163, 140), with (162, 151), she (151, 152), on (141, 175), the (138, 88), and to (124, 73). 3.2
The task was open for everybody. We particularly welcomed academic and industrial researchers, scientists, engineers and graduate students in SR, natural language processing, and biomedical/health informatics. We also encouraged participation by multi-disciplinary teams that combine technological skills with nursing expertise.
By 30 April 2015, 48 people had registered their interest in the task through the CLEF 2015 registration system,18 and 21 of these team leaders had emailed to confirm their participation. From its opening on 27 October 2014 to this date, the task discussion forum gained five other members than the organizers.19
By 1 May 2015, two teams submitted four methods. The first team, called TUC_MI/MC, was from the Technische Universität Chemnitz (TUC) in Germany. Its members were two researchers from the field of computer science, supervised by two TUC professors. They followed an interdisciplinary approach where one part brought the expertise from the field of speech processing to develop strategies for web-based language model adaptation. The other one came from the field of information retrieval to choose and develop methods for selecting and processing web resources to build a thematically coherent adaptation corpus. The second team, called UC, was from the University of Canberra in the Australian Capital Territory. It consisted of two PhD 17 http://www.nicta.com.au/nicta-synthetic-nursing-handoveropen-data-software-and-demonstrations/. 18 http://clef2015-labs-registration.dei.unipd.it/ 19 https://groups.google.com/forum/#!forum/clefehealth2015-task1a-speech-recognition students and three Professors from multi-disciplinary backgrounds, including clinical, public health, machine learning, and software engineering, working in collaboration.20
TUC_MI/MC submitted two SR methods. Their approach assumed each document having its own context and hence suggested adapting SR for each document separately. They used a two-pass decoding strategy: First, a verbal document was speech recognized. Then, keywords of the utterances were extracted and used as queries in order to retrieve web resources as adaptation data to build a document-specific dictionary and language model with the interpolation weights of 0.8 and 0.9 for TUC_MI/MC.1 and TUC_MI/MC.2, respectively. Finally, re-decoding of the same document was performed using the adapted dictionary and language model.
Also UC submitted two SR methods. UC.1 was based on acoustic modeling of speech using Hidden Markov Models (HMM). The verbal documents were preprocessed, including filtering, word level segmentation, and Mel Frequency Cepstral feature extraction, and HMM models were built for the training data. As there were no repetitions of data from different sessions, bagging and bootstrapping of training data were used. UC.2 combined language and acoustic models using the CMU Sphinx open source toolkit for SR. A custom dictionary and language model was developed for the speaker of the training set, because none of existing dictionary and language models was suitable for her accent. Unfortunately, the organizers had to reject this second method as even after an update request and deadline extension to 10 May 2015, this submission failed to meet the evaluation criteria for the format and completeness. 3.3
The Dragon baseline clearly had the best performance (i.e., E = 38.5) on the independent set for testing, followed by the TUC_MI/MC.2 (E = 52.8), TUC_MI/MC.1 (E = 52.3), UC.1 (E = 93.1), and the Majority baseline (E = 95.4) (Table 1). The resubstitution performance of the first three methods was approximately the same (i.e., from E = 55.0 ± 0.9), but last two methods had nearly 100 per cent error also on the training set (Table 2).
The performance of the Dragon baseline on the test set was significantly better than that of the second-best method (i.e., TUC_MI/MC.2, W = 302.5, p < 10–12). However, this rank-2 method was not significantly better than the third-best method (i.e., TUC_MI/MC.1), but this rank-3 method was significantly better than the fourthbest method (i.e., UC.1, W = 0, p < 10–15). Finally, the performance of the lowestranked method (i.e., the Majority baseline) was significantly worse than that of this rank-4 method (W = 1,791.5, p < 0.05). 20 Including Prof LH, task co-leader, as an advisor who encouraged participation without any engagement in the team’s method experimentation and development. As noted in Section 2.2, he did not develop the test set nor had an access to it before other participants. We conclude the paper by comparing the results with prior work, validating the released data, and discussing the significance of this study.
SR at its best can achieve an impressive C of 90–99 with only 30–60 minutes of
personalization or adaptation to a given clinician’s speech [
We have studied correcting SR errors through post-processing in [26]. This approach is unsupervised and applies phonetic similarity to substituted words. Its evaluation on the 100 training documents gives promising results; in 15 per cent of all 1,187 unique substitutions by the Dragon baseline, the speech-recognized word sounds exactly the same as its reference word and 23 per cent of them are at least 75 per cent similar.
A basic scientific principle of the reproducibility of the results relies on availability of
open data, open source code, and open evaluation results [30, 31]. Access to research
data also increases the returns from public investment in this area; encourages
diversity of studies and opinion; enables the exploration of new topics and areas; and
reinforces open scientific inquiry [32]. Whilst this open movement in health sciences and
informatics is progressing, particularly for source code [33] and evaluation results
from clinical trials [
However, data de-identification on text documents is fraught with difficulties [
Due to the lack of existing text corpora, that match the Australian clinical setting,
and due to the difficulty of providing ethically-sound open data, we have
compromised by providing synthetic data that closely matches the real data typically found in
a nursing shift-change. We have validated this matching by employing and
projectfunding clinical experts to confirm the typicality and compare the synthetic
documents with real data and related processing results [
The significance of our open synthetic clinical data lies in supporting innovation and
decreasing barriers of method research and development. In particular for new
participants in SR, IE, and other automated generation or analysis of text documents, the
aforementioned barrier of data access costs money and time. The entry and
transaction are even more expensive, but given the required expertise in the field, only the
latter cost can be decreased substantially by simplified licensing through open data
movement [
This shared task was partially supported by the CLEF Initiative and NICTA, funded by the Australian Government through the Department of Communications and the Australian Research Council through the Information and Communications Technology (ICT) Centre of Excellence Program. We express our gratitude to Maricel Angel, Registered Nurse at NICTA, for helping us to create this dataset. Last but not least, we gratefully acknowledge the participating teams’ hard work. We thank them for their submissions and interest in the task. 19. Ayres, R. U.., Martinás, K.: 120 wpm for very skilled typist. In: On the Reappraisal of Microeconomics: Economic Growth and Change in a Material World, p. 41. Edward Elgar Publishing, Cheltenham, UK & Northampton, MA, USA (2005) 20. Allan, J., Aslam, J., Belkin, N., Buckley, C., Callan, J., Croft, B., Dumais, S., Fuhr, N., Harman, D., Harper, D. J., Hiemstra, D., Hofmann, T., Hovy, E., Kraaij, W., Lafferty, J., Lavrenko, V., Lewis, D., Liddy, L., Manmatha, R., McCallum, A., Ponte, J., Prager, J., Radev, D., Resnik, P., Robertson, S., Rosenfeld, R., Roukos, S., Sanderson, M., Schwartz, R., Singhal, A., Smeaton, A., Turtle, H., Voorhees, E., Weischedel, R., Xu, J., Zhai, C.: Challenges in information retrieval., language modeling: Report of a workshop held at the Center for Intelligent Information Retrieval, University of Massachusetts Amherst, September 2002. SIGIR Forum 37(1), 31–47 (2003) 21. Kuniavsky, M.: Observing the User Experience: A Practitioner’s Guide to User Research.
Morgan Kaufmann Publishers, San Fransisco, CA, USA (2003) 22. Australian Government, Department of Health. Chronic disease: Chronic diseases are leading causes of death., disability in Australia, http://www.health.gov.au/internet/main/publishing.nsf/Content/ chronic (last updated: 26 September 2012) 23. Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bulletin 1(6), 80–83 (1945) 24. Shapiro, S. S., Wilk, M. B.: An analysis of variance test for normality (complete samples).
Biometrika 52(3–4), 591–611 (1965) 25. Razali, N., Wah, Y. B.: Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors., Anderson-Darling tests. Journal of Statistical Modeling., Analytics 2(1), 21–33 (2011) 26. Suominen, H. and Ferraro, G.: Noise in speech-to-text voice: Analysis of errors and feasibility of phonetic similarity for their correction. In Karimi, S. and Verspoor, K. (eds.) Proceedings of the Australasian Language Technology Association Workshop 2013 (ALTA 2013), pp. 34– 42. Association for Computational Linguistics (ACL), Stroudsburg, Brisbane, QLD, Australia. (2013) 27. Powers, D. M. W. Applications and explanations of Zipf’s law. In Powers, D. M. W. (ed.): Proceedings of the Joint Conferences on New Methods in Language Processing and Computational Natural Language Learning (NeMLaP3/CoNLL’98), pp. 151–160. ACL, Stroudsburg, PA, USA (1998) 28. Devine, E. G., Gaehde, S. A., Curtis, A. C.: Comparative evaluation of three continuous speech recognition software packages in the generation of medical reports. JAMIA 7(5), 462– 468 (2000) 29. Al-Aynati, M. M. and Chorneyko, K. A.: Comparison of voice-automated transcription and human transcription in generating pathology reports. Achieves of Pathology and Laboratory Medicine 127(6), 721–725 (2003) 30. Sonnenburg, S., Braun, M. L., Ong, C. S., Bengio, S., Bottou, L., Holmes, G., LeCunn, Y., Müller, K.-R., Pereira, F., Rasmussen, C. E., Rätsch, G., Schölkopf, B., Smola, A., Vincent, P., Weston, J., Williamson, R. C.: The need for open source software in machine learning.
Journal of Machine Learning 8, 2443–2466 (2007) 31. Pedersen, T.: Empiricism is not a matter of faith. Computational Linguistics 34(3), 465–470 (2008) 32. Organisation for Economic Development (OECD): OECD Principles and Guidelines for
Access to Research Data from Public Funding. OECD, Danvers, MA, USA (2007) 33. Estrin, D. and Sim, I.: Open mHealth architecture: An engine for health care innovation. Science 330(6005), 759–760 (2010)
The command bin/sclite -r reference.txt -h reference.txt trn -i spu_id -o all should produce perfect results by using the formatted reference standard both as a reference standard (-r reference.txt) and speech-recognized (or hypothesized -h) text. The command bin/sclite -r reference.txt -h dragon_nursing.txt -i spu_id -o all uses the formatted reference standard and formatted Dragon output, and hence should produce the aforementioned document-specific and overall evaluation results. The -i spu_id option refers to the transcription (TRN, i.e., a TXT file where each paragraph captures a handover document, followed by (documentID_V) with _V specifying the person whose voice/speech is recognized) formatted input files (with the default extended American Standard Code for Information Interchange encoding and the default GNU diff alignment) to pair the reference standard with the speech-recognized documents. The -o all option results in not only the evaluation results as percentages and raw numbers but also more details for analyzing correctly and incorrectly detected text patterns.
Removing punctuation and changing to ASCII