In this working notes paper we present our methodology in clinical speech recognition for the Task 1.a.1 of the CLEF eHealth Evaluation Lab 2015. The goal of this task is to minimize the worddetection errors. Our approach is based on the assumption that each spoken clinical document has its own context. Hence, the recognition system is adapted for each document separately. The proposed method performs two-pass decoding whereas the rst transcript is processed to queries which are used for retrieving web resources as adaptation data to build a document-speci c dictionary and language model. The second pass decodes the same document using the adapted dictionary and language model. The experimental results show a reduction of the insertion errors in comparison to the baseline system, but no improvement of the overall incorrectness percentage across all spoken documents.
In general, the creation of acceptable transcripts of spoken language requires high
human intervention and remains time- as well as cost-intensive. Since manual
generated transcriptions are a challenging task, especially for large and
heterogeneous datasets, it is more appropriate to apply automatic speech recognition
(ASR). In the medical domain, ASR supports a typical handover work ow as a
rst step by transforming verbal clinical information into electronic structured
records. The CLEF eHealth Evaluation Lab 2015 [
In this connection, out-of-vocabulary (OOV) has a serious impact on ASR
results. It necessarily requires the utilization of domain-speci c language models
(LMs) in order to cope with the huge amount of data and di erent topics. For
this purpose, the adaptation of a generic LM with a more speci c LM using
weighted linear interpolation is a common way. Supervised LM adaptation is
very costly for huge amount of data and di erent topics, since the generation of
speci c corpora takes a lot of time. A conclusive way is an unsupervised method,
which takes the context of a situation into account. As described in [
In this working notes paper we present our methodology and the results we
obtained in Task 1.a.1 of the CLEF eHealth Evaluation Lab 2015. Our approach
is based on the assumption, that each spoken clinical document has its own
context. Therefore, we suggest adapting ASR for each document separately. The
proposed method uses a two-pass decoding strategy. First, the transcript of
a document is generated by an ASR system. Keywords of the utterances are
extracted and used as queries in order to retrieve web resources as adaptation
data to build a document-speci c dictionary and LM. Finally, re-decoding of
the same document is performed using the adapted dictionary and LM. The
developed system was already applied in the previous works [
This Paper is organized as follows: In the next section we present our method for unsupervised language model adaptation in clinical speech recognition. In Section 3 we describe the applied dataset, the experimental setup, and the evaluation results. Finally, we conclude this paper in Section 4 and give some future directions. 2
Our method works out-of-the-box with a two-pass decoding strategy. First, a transcript of utterances in the spoken document is generated by ASR. The segmentation of the transcript into several units is performed by the recognizer itself using long silences. Each segment ranges from a short statement to a whole sentence. The segments are processed and used as queries for retrieving adaptation data to build a document-speci c dictionary and LM. The second pass of the recognizer decodes the same document using the adapted dictionary and LM. As shown in Fig. 1, a transcribed segment generated by the ASR system is used for building a query in order to perform a web search. Since a segment often contain more words than useful for a web search query, especially for retrieving documents in a close context, the following steps are performed to limit their number: 1. Nouns, plural nouns and the corresponding adjectives are extracted to obtain the most meaningful words. 2. A pre-de ned stop-word list is applied which is derived from the training data and contains unnecessary as well as recurring vocabulary (e.g., date and time speci cation) 3. If 2. yields more keywords than a prede ned threshold, the sequence of words is split into several parts with almost the same number of words ful lling the requirements and each of these parts is considered to be a separate query. Otherwise there is only one query.
For each resulting query a web search is conducted. The amount of the retrieved web documents is combined and normalized before adding to the adaptation corpus. In detail, the pure articles of the retrieved web documents are extracted and special characters, acronyms and numbers are converted in order to be conform to the conventions of the pending adaptation process and the ASR system. These steps are performed for all transcribed segments of one spoken document and their normalized texts are accumulated to one corresponding adaptation corpus. 2.2
The accumulated adaptation corpus is used for modifying the base dictionary and the base LM as illustrated in Fig. 2. The pronunciation dictionary adaptation aims to enrich a base dictionary by new vocabulary coming from the adaptation corpus. For this purpose, the vocabulary of the corpus is extracted and compared to the base dictionary. The additional vocabulary is phonetically transcribed by a grapheme-to-phoneme (G2P) decoder and combined into a temporary dictionary. Finally, the temporary and the base dictionary are merged to an adapted dictionary.
The LM adaptation is performed by a weighted linear interpolation of the temporary and the base LM. The temporary LM is trained by means of the Vocabulary Extraction &
G2P Dictionary Merging Adapted Dictionary
Train LM
Linear Interpolation
Adapted
LM
Base Dictionary Base LM adaptation corpus, whereas the base LM is a more general model trained on topic-independent data collections. Finally, the vocabulary of the resulting model is a superset of the vocabulary of both, the temporary and the base LM. 3
Before describing the details of the experimental setup, the used dataset is introduced and some observations on conducted preliminary experiments are stated. Afterwards we discuss our experimental intermediate as well as nal results of the evaluation. 3.1
In this work the NICTA Synthetic Nursing Handover Data dataset [11] is used which was created at NICTA in 2012-2014 for clinical speech recognition and information extraction related to nursing shift-change handover. The training set as well as the test set consist of 100 written, free-form text documents and the corresponding recorded audio les spoken by an Australian registered nurse with over twelve years of working experience. The text documents of the test set were not released for evaluation purposes. Furthermore, this dataset includes recordings lasting about half an hour of her reading an excerpt of \The Final Odyssey" as initialization data for speech recognition engines.
In a preliminary experiment the ASR output of the training set was compared to the written, free-form text documents. This exposed some common errors, which are partially already mentioned in the task description. Further investigation revealed, that there are some abbreviations used instead of the correctly spelled words. However, these words are verbalised correct by the nurse in the provided audio les. Therefore, a list of usual misspellings was created, which should be used instead of the correct words generated by the standard con guration of the ASR system. For instance, only 26 appearances of the correctly spelled word \years" were counted, but 53 appearances of the abbreviation \yrs". Using the abbreviation should lead to less substitutions counted for the nal evaluation metric. More detected substitutions are shown in Table 1. Beside these misspellings in the text documents, there are some misspeaks and following corrections by the nurse as well as some ller words like \ehm". Another observation is the usage of numerals and numbers in the written, free-form text documents. Only 22 numerals from \one" to \eight" are used but 266 numbers up to three digits (more than half of this with only one digit) were found. Hence, always using numbers instead of numerals seems promising. 3.2
ASR was performed using the engine of the open-source framework sphinx-4 [12]. As a basic con guration, we used already existing components. We applied the acoustic model \HUB4" (http://www.speech.cs.cmu.edu/sphinx/models/), which has been trained using 140 hours of 1996 and 1997 hub4 training data. It includes 3-state within-word and cross-word triphone Hidden-Markov-Models with 8 Gaussian mixture models. We performed maximum a posteriori (MAP) adaptation by using the initialization data of the training set to update the parameters of the acoustic model in order to better match the observed data. Next, we used the pronunciation dictionary \cmudict.0.7a SPHINX 40" (https://svn.code.sf.net/p/cmusphinx/code/trunk/cmudict/sphinxdict/), which comprises 133k words and the corresponding phonetic transcription. We modied the dictionary concerning the notation of the clinical reports, for instance, adding new vocabulary or replacing words with abbreviations. Moreover, we assigned some vocabulary to ller words to avoid misinterpretation caused by llers (e.g., \ah" or \ehm"). As a generic LM we used the \US English Generic Language Model" (http://sourceforge.net/projects/cmusphinx/ les/) comprising about 3.2M trigrams. The utilization of the generic LM on the free-form text documents of the training set results in a perplexity of 324.8 and 1.1k OOV words. In our experiments language modeling was conducted with the SRILM toolkit [13]. We performed LM adaptation using the generic LM and the freeform text documents of the training set in order to obtain our base LM for the proposed method. Our goal was to generate a background model which has the properties of clinical documents as well as an appropriate generalization. Hence, we assigned equal interpolation weights.
Concerning the proposed adaptation method, the temporary LMs were constructed as trigram models using Kneser-Ney smoothing. These models were combined with the base LM by means of a weighted linear interpolation in order to perform the LM adaptation. We used the WFST-driven G2P framework Phonetisaurus [14] to phonetically transcribe temporary dictionaries. For this purpose, we trained a G2P model based on 133k words from the applied pronunciation dictionary, which works stable for typical English words.
The accumulation of the adaptation corpus was achieved by parsing the website of the Journal of Postgraduate Medicine (http://www.jpgmonline.com). To accomplish this, a segment of the rst-pass transcript is processed by the Stanford Lexical Parser [15] to extract keywords. We assigned a threshold of 5 for the keyword extraction, i.e., if there are more than ve words left, the sequence of words is split into separate parts with an almost equal number of words by trying to keep adjectives and their corresponding nouns together. Considering the next noun belonging to an adjective results in parts of up to seven words. Each part is considered to be a separate query which is utilized in the search function of the web portal. The resulting list of full-text articles was prioritized concerning a relevance of 50% and higher. We limited the maximum number of retrieved articles to 100. 3.3
The intermediate results of our adaptation method are relevant for further processing steps and consequently for the nal results. Table 2 gives an overview of the mean values concerning the retrieval process across all spoken documents. Comparing the numbers of segments per spoken documents leads to the conclusion, that the speech recognizer was able to detect much more pauses in the training set than in the test set. Fewer segments in the test set also lead to fewer but longer queries built by the system, as seen in line two and three of Table 2. As the number of web documents retrieved per query is almost equal for both datasets, the number of web documents per spoken document is also much higher for the training set than for the test set.
The number of tokens per web document (after normalization) are quite similar. Hence, the di erences in the resulting adaptation corpora for the training and the independent test set, as shown in Table 3, can only be traced back to the di erences in the number of segments per spoken document. The statistics in Table 3 indicate that more tokens in the adaptation corpus lead to more types and also to more new types, which were used to extend the dictionary of the ASR system for the second pass.
A series of speech recognition experiments was carried out using the two di erent interpolation weights =0.8 and =0.9 for the proposed adaptation method on the training set and the independent test set. These weights imply a higher preference of the base LM which was designed for the clinical freeform text documents. The results are illustrated in Table 4. The baseline results were achieved by Dragon Medical 11.0 which was trained on the initialization data and applied with the vocabulary of nursing. In general, it can be seen that for both datasets our system had more substitution errors and deleted some more words than the baseline system. We considered the speci c notation of the written, free-form text documents such as abbreviations or llers, which leads to less inserted words than the baseline system. Compared to the baseline, the mean value of the incorrectness percentage across all documents in the training set was improved by our system with 2.1% ( =0.9). The reason for that is the consideration of the speci c notation of the written, free-form text documents and the adaptation of the generic LM using the free-form text documents of the training set which increases the probability of recognizing the correct words.
The performance measurements of our system on the test set in comparison to the training set concerning the mean of the incorrectness percentage across all documents show similar results with a decrease of 3.6% ( =0.9). Our system generated many substitution errors on the test set with 36.6% and a di erence of 14.0% to the baseline that is crucial for the overall incorrectness percentage. We achieved a mean value of 52.1% and consequently 12.6% over the baseline. However, we could achieve a reduction of the insertion errors with 5.1% ( =0.8) and 5.0% ( =0.9). All in all, the evaluation on the test set shows that our system did not improve the baseline mean incorrectness percentage. The con guration with the interpolation weight =0.9 was just slightly better than the lower one. 4
We presented a method for unsupervised language model adaptation in automatic speech recognition for the Task 1.a.1 of the CLEF eHealth Evaluation Lab 2015. Our approach is based on the assumption, that each spoken clinical document has its own context. Hence, the recognition system is adapted for each document separately. The method uses a two-pass decoding strategy, whereas the rst transcript is processed to queries, which are used for retrieving web resources as adaptation data to build a document-speci c dictionary and language model. The second pass of the speech recognizer decodes the same document using the adapted dictionary and language model. The experimental results on the test set showed a reduction of the insertion errors in comparison to the baseline system. We achieved a mean value of 52.1% incorrectness across all documents. All in all, we did not improve the baseline incorrectness percentage, since our system produces more substitution errors and deleted some more words. The con guration of our method with the interpolation weight =0.9 was just slightly better than =0.8.
However, further improvements could be achieved by a more sophisticated selection of the retrieved adaptation data. For instance, a model for disease classi cation in text corpora could be helpful to obtain only speci c adaptation data for the corresponding spoken document. Moreover, it would be interesting to use further resources from web, like Twitter and RSS Feeds. For future work, the investigation of phonetics for accented speech and consequently the application of pronunciation modeling should enhance the performance of the recognition system.
Acknowledgments. This work was partially funded by the German Federal Ministry of Education and Research within the project MACeLot (funding code 16SV7260) and the program of Entrepreneurial Regions InnoPro le-Transfer in the project group localizeIT (funding code 03IP608X). 11. Suominen, H., Zhou, L., Hanlen, L., Ferraro, G.: Benchmarking clinical speech recognition and information extraction: New data, methods, and evaluations. JMIR Medical Informatics. (2015) 12. Walker, W., Lamere, P., Kwok, P., Raj, B., Singh, R., Gouvea, E., Wolf, P., Woelfel, J.: Sphinx-4: A Flexible Open Source Framework for Speech Recognition. Technical Report. Sun Microsystems, Inc., Mountain View, CA, USA. (2004) 13. Stolcke, A., Zheng, J., Wang, W., Abrash, V.: SRILM at sixteen: Update and outlook. In: Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop, p.5. (2011) 14. Novak, J. R.: Phonetisaurus: A wfst-driven phoneticizer. The University of Tokyo,
Tokyo Institute of Technology, pp.221{222. (2011) 15. Klein, D., Manning, C. D.: Fast Exact Inference with a Factored Model for Natural Language Parsing. In: Advances in Neural Information Processing Systems 15 (NIPS 2002), Cambridge, MA: MIT Press, pp. 3{10. (2003)