CLEF eHealth 2013 Task 1 requires participants to perform named entity recognition and normalization of disorder mentions from clinical reports, where two important questions need to be addressed: (a) discovering mentions of concepts that belong to the UMLS semantic group Disorders, and (b) mapping each mention to a SNOMED-CT concept represented by a Concept Unique Identifier (CUI). The disorder mentions here could be a single text span (e.g. "loss of consciousness") or multiple text spans (e.g. "right ventricular" ... "dilated"). The corresponding concepts are usually encoded in SNOMED-CT, but sometimes may not be formally defined. To tackle these challenges we designed a two-stage annotation system, where MetaMap serves as the first-stage annotator for the identification of phrases with potential interest, and a Rulebased annotator works on the second stage for fine-grained supplementation. MetaMap has mature technical features and relatively good performance on medical text analysis. It is developed to link the text of medical documents to the knowledge embedded in UMLS Metathesaurus. Highly configurable on semantic types, it enables us to specify the output concepts of interest. However, MetaMap is incapable of mapping text to concepts undefined, which, unfortunately, is the case for a number of disorder mentions in the task. Thus, we proposed a rule-based approach as the second-stage annotator. The annotation rules are learned from errors MetaMap made on training data, and could successfully recognize those undefined concepts. We also proposed Normalization and Postprocessing algorithms to normalize and prune the intermediate results for better matching. The experiments on training data demonstrate the effectiveness of every system component. MetaMap fails on pinpoint identification, but has certain capability to parse and roughly recognize the phrases of interest. Once combined with normalization, it could attain 0.463 F-score on training data. Designed to correct false negative errors, the individual Rule-based annotator is able to identify up to 15% all true annotations. The entire system eventually achieves 0.68 F-score in Task 1a and 0.57 accuracy in Task 1b. From the final competition results, our system performs consistently on test data, and beats all other participating systems in the group with additional annotations.
Clinical reports, such as discharge summary, radiology reports, echocardiogram
reports and electrocardiograph reports, are abundant in mentions of clinical conditions,
anatomical sites, medications, and procedures, which is in stark contrast with the
newswire domain where text is dominated by mentions of countries, locations and
people. Many surface forms are representations of the same concept. Unlike the
general domain, in healthcare area there are rich lexical and ontological resources that
can be leveraged when building applications. The Unified Medical Language System1,
(UMLS) represents over 130 lexicons/thesauri with terms from a variety of languages.
The UMLS Metathesaurus integrates resources used worldwide in clinical care, public
health, and epidemiology, including SNOMED-CT2, ICD-93, and RxNORM4. In
addition, the UMLS also provides a semantic network in which every concept in the
Metathesaurus is represented by its Concept Unique Identifier (CUI) and is semantically
typed [
Because the recognition and normalization of named entity mentions is a
fundamental task, it becomes the focus of CLEF eHealth 2013 Task 1 [
“atrial fibrillation” is a mention of type Disorders with CUI C0004238. UMLS preferred term is “atrial fibrillation”. 2. The left atrium is moderately dilated.
“left atrium.... dilated” is a mention of type Disorders with CUI C0344720. UMLS preferred term is “left atrial dilatation”. 3. 53 year old man s/p fall from ladder.
“fall from ladder” is a mention of type Disorders with CUI C0337212. UMLS preferred term is “accidental fall from ladder”. 4. The patient was admitted with low blood pressure.
“low blood pressure” is a Finding in UMLS, and as such does not belong to the definition of the Disorder semantic group. In this case, however, because it does indeed describe a disorder, it should be annotated. The CUI is left empty as “CUIless”.
Example 1 above represents the easiest cases. Example 2 represents mentions that are disjoint. Example 3 is a synonym of the UMLS preferred term. Example 4 represents mentions that have no corresponding mapping concepts in UMLS.
The scope of current task is limited to clinical reports written in English language, with the normalization/mapping to SNOMED-CT CUIs in ULMS version 2011AA. Illustrated by the above examples, Task 1 requires us to solve two problems: (a) discovering the boundaries of disorder mentions, and (b) mapping each mention to a SNOMED-CT concept. The system output should contain both boundaries and CUIs information. In light of running the given evaluation code, every disorder annotation should follow the format below.
report name || annotation type || cui || char start || char end 00176-102920-ECHO_REPORT.txt||Disease_Disorder||C0031039||120||140 If the annotation contains disjoint spans (i.e., non-contiguous spans, such as in the sentence "Abdomen: no distention is noted." in which the single annotation for "abdominal distention, C0235698" encompasses the span 0-6 (abdomen) and 13-22 (distention)), additional character start and character end values of every following span will be appended to those of the first. 00176-102920-ECHO_REPORT.txt||Disease_Disorder||C0344720||430||441||456||463 2
We designed a two-stage annotation system to tackle Task 1 (see Fig.1). At the first stage, MetaMap is employed to parse clinical reports and annotate disorder mentions. However, constrained by the accuracy of MetaMap and the special usage of some terms (e.g. Example 4), not all disorder mentions can be precisely recognized. Especially, all words/phrases that should be mapped to a “CUI-less” concept will not be annotated by MetaMap. Therefore, we proposed a second-stage rule-based annotation in our system to supplement MetaMap. Annotation rules are learned from two types of errors made by MetaMap on training data. Besides, for better matching, the intermediate results are normalized and pruned in Normalization and Post-processing steps. The functionality of system components will be detailed below individually. 2.1
We chose MetaMap as the first-stage annotator for three reasons. Firstly, MetaMap
has mature technical features and relatively good performance on medical text
analysis. It is developed to link the text of medical documents to the knowledge embedded
in UMLS Metathesaurus. MetaMap employs a knowledge-intensive approach, NLP,
and computational-linguistic techniques [
Secondly, because disorder mention is defined as span(s) of text that belongs to the Disorder semantic group, we need to limit the scope of annotation on text from that semantic group only. MetaMap is highly configurable on semantic types of concepts and thus enables us to specify the output of interest. To be specific, we restricted the annotation from one of the following ULMS semantic types:
Congenital Abnormality Injury or Poisoning Disease or Syndrome Cell or Molecular Dysfunction Anatomical Abnormality Signs and Symptoms
Acquired Abnormality Pathologic Function Mental or Behavioral Dysfunction Experimental Model of Disease Neoplastic Process
Thirdly, MetaMap has been broadly used in industry and academia. It can be treated as a benchmark and foundation to compare different algorithms and fulfill advanced analytics built over it.
Clinical Reports
MetaMap Annotation
Normalization
Training
Data Rule-based Annotation
Postprocessing
System
Output The output of MetaMap is the phrase from shallow parsing with its corresponding concept in one of semantic types specified above. Table 1 shows three records extracted from MetaMap output. Record 1 is a perfect matching because the phrase string and concept string are identical. Then the boundaries of this disorder mention can be determined directly from the position of the phrase “Pericardial effusion” in the original report. Record 2 and 3, however, are cases that only part of the phrase string matches its concept string. In such scenario we need to normalize the phrase by virtue of the concept. Basically we only keep the words that appear in both phrase and concept. Thus, “left ventricular hypertrophy” and “SOB” will be the new annotation in Record 2 and 3, respectively. MetaMap is capable of mapping text to existing concepts encoded in SNOMED-CT, but fails if corresponding concepts are not formally defined. Unfortunately, a portion of disorder mentions in Task 1 belong to the latter case, which enlightened us to propose a second-stage annotation for improved performance. We focused on the false negative errors, the true annotations missed by MetaMap, by comparing its output with the gold standards of training data, and then created corresponding rules to match the words/phrases of interest. To avoid overfitting and tune system performance, after applying a rule, we inspected the ratio of the size of its true annotations over the size of its false annotations, and set a threshold to control whether this rule should be included or not.
#( #( ) ) ≥ (1)
Basically the proposed annotation method is composed of three types of rules with regard to the style of required regular expressions, which are encoded to catch strings with certain patterns.
Single Span. The majority of disorder mentions are expressed in a single text span, such as Example 1, 3, 4 in Section 1. Thus the annotation rule can be written in regular expression simply using the text itself (see Table 2). Multiple Spans. A small number of disorder mentions contain two or more text spans, such as Example (2) in Section 1. Thus the annotation rule needs a generic expression to capture all variants, which is usually achieved by using metacharacter in regular expression (see Table 3). Similar to the normalization of the MetaMap output, phrases identified in Rule-based annotation step also need refinement. The normalization in post-processing removes stopwords, quantitative values, and descriptive words from annotations, such as “any”, “severe” and “obvious”. The final step is annotation pruning, which reduces Acronyms and Abbreviations. Acronyms and abbreviations are used extensively in clinical notes. They are convenient shorthands in writing records, instructions, and prescriptions, and space-saving devices. Efforts have been made to standardize the form of them in some journals and books, but they generally vary from person to person. Learned from training data, frequent acronyms and abbreviations are linked to theirs CUIs by matching the entire word (see Table 4). We add the anchor metacharacter “\b” in regular expression to match the word boundary, so that, for example, only the word “MR” will be matched by “/\bMR\b/”, instead of the word “COMRADE”. the false positive errors by filtering out phrases that match any rules on a blacklist. This blacklist is learned from training data set by analyzing the false positive annotations given by MetaMap. 3
Task 1 provides a training data set and a withheld test data set. The training data contain 199 clinical reports with 5238 disorder mentions annotated, where rules are learned and parameters are tuned The test data contain another 100 reports with 4513 disorder annotations for evaluation purpose only. As mentioned earlier, Task 1 requires participants to solve two problems: (a) identifying the boundaries of disorder mentions and (b) mapping each mention to a SNOMED-CT concept. We will report our experimental results on these two subtasks separately, and in each task a strict evaluation and a relaxed evaluation are run individually. The strict evaluation requires the annotated text span to be identical to the reference standard span, while the relaxed evaluation only requires the annotated text span has overlap with reference standard span. 3.1
We conducted a series of experiments on training data to evaluate the effectiveness of the system components we proposed. Table 5 summarizes the performance of different system components and their combinations in two subtasks. Since the system is tuned and optimized for F-score under the strict evaluation standard, we will discuss and compare F-scores under this standard below, unless otherwise noted. STtarsickt1b Accuracy 0.059 0.391 0.3 0.144 0.526 0.544 0.551 TRaeslakx1ead Accuracy 0.782 0.874 0.781 0.956 0.897 0.873 0.9 System Components: 1 – MetaMap annotation, 2 – Normalization, 3 – Rule-based annotation, 4 – Post-processing including Normalization (4.1) and Pruning (4.2).
Comparing Comp. 1 and Comp. 1+2 in Table 5, we can see (1) MetaMap as a standalone annotator delivers quite poor results; (2) however, it has much better score in relaxed evaluation, and out of its raw output, near half of correct annotations can be obtained after normalization. This indicates that MetaMap fails on pinpoint identification, but has the capability to parse and roughly recognize the phrases of interest. Therefore, despite an unreliable tool to complete the work individually, MetaMap could serve as a reasonable platform for upper level algorithm development.
Designed to correct false negative errors, the Rule-based annotation alone (Comp. 3) is able to identify up to 15% of all true annotations. Working with Comp. 1+2, the combination attains a 0.549 F-score.
The post-processing step is also very important. Its normalization (4.1) and pruning (4.2) algorithms give 0.05 and 0.08 F-score lifts over Comp. 1+2+3, respectively. Unifying all components, the entire system eventually achieves 0.68 F-score in Task 1a and 0.57 accuracy in Task 1b. 3.2
Participants are allowed to submit two runs of annotations on test data for each subtask. Systems using additional annotations (Group B) will be evaluated separately from systems without additional annotations (Group A). Our system is in Group B, since we employed MetaMap for the first-stage annotation. The competition organizers published the final results and team rankings for each group in each subtask online5. For easily reading and comparing, we compiled all teams together in Table 68 using the field “Group” for differentiation. The best result out of the two submissions is taken for each team. The ranking is based on F-score in Task 1a and accuracy in Task 1b.
From Table 6-8 we are glad to see that our system (UCSC) has consistent perfromance in every subtask on test data, and it outperforms all other participating systems in Group B in every subtask under either strict or relaxed evaluation standard. Even if we compare with all other participants, ignoring the group difference, our system successfully ranks 4th and 3rd in subtask 1a and 1b respectively. This achievement encourages the broad MetaMap users who contemplate a relatively high performance annotator built on top of MetaMap without spending much effort on NLP infrastructure.
Though the detail of the leading systems has not been published yet, we guess the gaps between our system and them are from three aspects: (1) probably MetaMap is not able to deliver annotations as precise and complete as those customized, advanced annotation systems; (2) with a portion of CUI-less concepts undefined, the vocabulary of SNOMED-CT is kind of limited for current task, resulting in a number of candidates unidentified by MetaMap; (3) the annotation rules learned from training data are difficult to capture certain features of the annotation, such as the sequential information among words, which may be supplemented by statistical learning algorithms. In future, we are going to advance our system along these directions. 5 http://nicta.com.au/business/health/events/clefehealth_2013/results
In this paper we proposed a two-stage annotation system to solve CLEF eHealth 2013 Task 1, where MetaMap serves as the first-stage annotator for the identification of phrases with potential interest, and a Rule-based annotator works on the second stage for fine-grained supplementation. Learned from training data, the annotation rules are generated to eliminate two types of errors from MetaMap. The experiments on training data demonstrate the effectiveness of every system component, while the published competition results show that our system performs consistenly on test data and beats all other competing systems in the group using additional annotations.
Our results in this paper are applicable to healthcare text mining and disorder
annotation from clinical reports. We anticipate that these results can be generalized further
and that their use can be extended into many new domains such as network design
diagnostics, semiconductor manufacturing, aerospace system operation, and
automotive system design. This expectation is based on prior related work by us in the
networks [
We wish to thank the organizers of the CLEF eHealth 2013 for preparing the datasets and organizing the shared tasks. Their work is supported by the Shared Annotated Resources (ShARe) project funded by the United States National Institutes of Health: R01GM090187. steven_seeger alamb RelAgent Rahul CORAL THCIB LIMSI KPSCMI SNUBME
FAYOLA
NCBI Mayo
UTHealth_CCB THCIB KPSCMI CLEAR alamb AEHRC steven_seeger UCDCSI Rahul Diganesan AEHRC ArvindWVU FAYOLA NCBI Group
A A B A A B A A B A A B A A A A A