The Australian e-Health Research Centre (AEHRC) recently participated in the ShARe/CLEF eHealth Evaluation Lab Task 1. The goal of this task is to individuate mentions of disorders in free-text electronic health records and map disorders to SNOMED CT concepts in the UMLS metathesaurus. This paper details our participation to this ShARe/CLEF task. Our approaches are based on using the clinical natural language processing tool Metamap and Conditional Random Fields (CRF) to individuate mentions of disorders and then to map those to SNOMED CT concepts. Empirical results obtained on the 2013 ShARe/CLEF task highlight that our instance of Metamap (after ltering irrelevant semantic types), although achieving a high level of precision, is only able to identify a small amount of disorders (about 21% to 28%) from free-text health records. On the other hand, the addition of the CRF models allows for a much higher recall (57% to 79%) of disorders from free-text, without sensible detriment in precision. When evaluating the accuracy of the mapping of disorders to SNOMED CT concepts in the UMLS, we observe that the mapping obtained by our ltered instance of Metamap delivers state-of-the-art e ectiveness if only spans individuated by our system are considered (`relaxed' accuracy).
The automatic identi cation of clinical conditions, such as disorders,
abnormalities, body sites, medications, procedures, devices, and their normalisation to a
standard terminology of reference, are tasks of key importance for the analysis
of free text electronic health records (e.g. discharge summaries). Solutions that
tackle these tasks are fundamental to unlock clinical information trapped in the
natural language of clinical narratives, which can be used to improve access,
reporting, reasoning and discovery. These capabilities have been, for example,
at the basis of previous research we conducted on cancer reporting [1{3],
radiology reconciliation [
Task 1 of the ShARe/CLEF eHealth Evaluation Lab aims to provide
researchers with a standard benchmark for evaluating clinical information
extraction and normalisation systems [
Details of this task can be found in the Lab overview paper [
To discover mentions of disorders in the free-text of discharge summaries
we implemented two solutions (runs TeamAEHRC.1 and TeamAEHRC.2) based
on Metamap[
Empirical results obtained in the ShARe/CLEF 2013 challenge suggest that the approach implementing CRF and Metamap is more e ective than using Metamap alone for identifying disorders from free-text discharge summaries. Our approach based on the combination of Metamap and CRF achieved considerably higher recall within the `relaxed' evaluation context than when considering the `strict' settings. This is because of a bug in the code we used for submitting our run. The bug produced a miss-alignment between the correct output of the CRF models and the text of the discharge summaries. Thus, the spans present in the submission representing this approach contain token misalignments, resulting in better e ectiveness when the `relaxed' settings are considered. Further considerations on the results of our approaches in this task are presented in section 3.2. 2
As part of our participation in this challenge, we rst investigated the e ectiveness of a system based on Metamap, and then the contribution that a supervised named entity recognition approach based on Condition Random Fields would have when added to the approach that relies solely on Metamap.
Metamap is a well-known software tool that uses natural language processing
and knowledge-intensive approaches to identify biomedical keywords and map
them to UMLS Metathesaurus concepts. We used Metamap as integrated within
AEHRC's Medtex medical text analysis platform [
Table 1 identi es which semantic types (as identi ed by Metamap) are most commonly associated with disorders (as identi ed by the expert assessors). To produce our rst submission to this year's ShARe/CLEF 2013, we use Metamap to identify spans of biomedical keywords and their associated UMLS concepts in the discharge summaries of the test set. To retain only spans that may refer to disorders, we lter out concepts that do not belong to the top 10 semantics types identi ed from the training dataset (i.e., we consider only the semantic types listed in Table 1 except Mental Process, Body Substance and Cell or Molecular Dysfunction). Normalisation is achieved by considering the CUIs of the resulting concepts as provided by Metamap. The submission to the ShARe/CLEF challenge that implements this method is identi ed as TeamAEHRC.1.
According to Table 1, a large number of disorders identi ed by expert assessors in the training discharge summaries are not identi ed by Metamap: in fact, 1,675 disorders have no semantic type. If the training dataset is representative of the testing dataset, a similar trend is likely to be observed when testing the previous approach on unseen data. This would then result in poor recall, as Metamap would miss a large number of mentions of disorders. With the objective of improving recall of our Metamap-based approach, we complete that solution with the use of a supervised machine learning model for name entity recognition. Speci cally, we chose to implement Conditional Random Fields (CRF) models to automatically identify spans of text that refer to disorders. We have used CRF models in previous work on de-identi cation of electronic health records, and we found that, provided enough training data is made available, CRF models are able to e ectively identify targeted named entities [13].
A Conditional Random Fields classi er is a discriminative undirected probabilistic graphical model that, given a observed sequence, de nes a log-linear distribution over labelled sequences. To build the CRF models we used the following lexical and semantic features: { the word tokens; { word shapes features (e.g. the presence of capitalised characters at the beginning of the word token or across the whole token); { letter n-grams (n = 6); { disjunctive features, which capture disjunctions of words and word shapes within windows of tokens; { position, which captures the position of a word in the sentence; { the UMLS CUIs as provided by Metamap; { a disorder ag as provided by our rst approach (i.e. TeamAEHRC.1).
Features were extracted from discharge summaries in the training and testing datasets. The CRF model was trained using discharge summaries from the ShARe/CLEF Task 1 training set only. The submission to the ShARe/CLEF challenge that implements this method is identi ed as TeamAEHRC.2. 3 3.1
To evaluate the e ectiveness of approaches to identify mentions of disorders in discharge summaries, the organisers of the ShARe/CLEF Task 1 considered: { Precision (P): TP / (TP + FP); { Recall (R): TP / (TP + FN); { F-measure (F): (2 * Recall * Precision) / (Recall + Precision); where true positive (TP) indicates that a system identi ed a disorder in the same span as that identi ed by the expert assessors, false positive (FP) refers to the identi cation of an incorrect span, and false negative (FN) indicates that a system failed to identify a disorder-span that was instead identi ed by the expert assessors. The `strict' and `relaxed' evaluation settings refer to the case where the automatically identi ed span is identical to the reference span, and that identi ed span overlaps with the reference span, respectively. We refer the reader to the task overview paper for more details.
Accuracy was chosen to evaluate the e ectiveness of approaches to map mentions of disorders in discharge summaries to SNOMED CT concepts in the UMLS. Accuracy is de ned as the ratio of correctly mapped concepts to the total number of mentions of disorders. In the `strict' evaluation settings, the total number of mentions of disorders is computed over the reference standard identi ed by the expert assessors. In the `relaxed' settings, the total number of mentions of disorders is computed over the mentions identi ed by the system that strictly overlap with the reference standard. 3.2
Results obtained by the two submitted runs are reported in Tables 2 and 3 for the identi cation of disorders and their mapping to UMLS concepts, respectively.
Results of TeamAEHRC.1 for task 1a (Table 2) suggest that our Metamap instance only identi es a limited number of disorders, with recall between 21.2% and 27.5%. While the relaxed evaluation setting does not sensibly a ect the recall e ectiveness of TeamAEHRC.1, it is observed that precision increases over the run evaluated within the strict setting. This suggests that if span overlaps between system annotation and reference standard are considered, the disorders identi ed by the system are highly likely to have been identi ed also by expert assessors. The high `relaxed' accuracy achieved by TeamAEHRC.1 on task1b (Table 2; indeed, the highest accuracy across the challenge systems) highlights that when spans of disorders are correctly identi ed, Metamap is highly e ective in providing a mapping consistent with those of the expert assessors.
Results of TeamAEHRC.2 submission are a ected by a bug in our code that prevents the correct alignment between positions of disorder annotations as identi ed by our system and the token positions in the challenge's reference standard. This explains why this system achieves signi cantly better e ectiveness across the `relaxed' evaluation settings than the `strict' one, both for task 1a and task 1b. The `relaxed' settings for task 1a account for partial overlaps between system annotations and reference standards; this partially corrects the bug in our system. The analysis of Table 1 suggested that the system based on Metamap may not identify a large number of disorders: this has proven to be the case (the recall of TeamAEHRC.1 is low). The intuition for complementing the Metamapbased approach with CRF models was that these may identify patterns in the text of discharge summaries that would allow to identify mentions of disorders Metamap would not recognise. The (relaxed) recall achieved by TeamAEHRC.2 suggests that this has been indeed the case: CRF models enable to identify about 3 times more mentions of disorders than the system based solely on Metamap (TeamAEHRC.1). The highest recall provided by CRF is traded o for a loss in precision (Precision of TeamAEHRC.1: 0.903; Precision of TeamAEHRC.2: 0.886): some of the spans identi ed by the system are actually not mentions of disorders according to expert assessors. However, this loss is minimal (1.88%) and indeed TeamAEHRC.2 obtains almost double the F-measure than that of TeamAEHRC.1 (TeamAEHRC.1: 0.422; TeamAEHRC.2: 0.833). 4
In this paper we have presented the methods used in our submissions to the ShARe/CLEF 2013 eHealth Evaluation Lab Task 1. Our methods are based on an instance of Metamap and on Conditional Random Fields. Empirical results suggest that if a disorders mention is correctly identi ed by Metamap, then its mapping to a CUI provided by this system is highy likely to be correct. However, Metamap does only recognise a handful of mentions of disorders: many of the disorders identi ed by the ShARe/CLEF expert assessors are not recognised by our instance of Metamap. To increase recall, we have complemented our Metamap solution with Conditional Random Fields models. Our implementation was a ected by a software bug, which prevented correct alignment of identi ng spans and tokens. The 'relaxed' evaluation settings partially addresses our span alignment issue. When 'relaxed' e ectiveness is considered, the solution that mixes Metamap and Conditional Random Fields (TeamAEHRC.2) is able to identify a large number of disorders, trading o only a small amount of the precision provided by Metamap. When considering mapping of free-text to a standard reference terminology (task 1b), the mapping provided by Metamap is found to be highly accurate for the mentions that have been correctly identi ed by the system. 12. Noemie Elhadad, W.C., O'Gorman, T., Palmer, M., Savova, G.: The share schema for the syntactic and semantic annotation of clinical texts (2013 (under review)) 13. Zuccon, G., Strachan, M., Nguyen, A., Bergheim, A., Grayson, N.: Automatic de-identi cation of electronic health records: An australian perspective. In: 4th International Workshop on Health Document Text Mining and Information Analysis (LOUHI'13). (2013) 1{5