The ShARe/CLEF eHealth Evaluation Lab (SHEL) organized a challenge on natural language processing (NLP) and information retrieval (IR) in the medical domain in 2013. The first task of the 2013 ShARe/CLEF challenge was to extract disorder mention spans and their associated UMLS (Unified Medical Language System) concept unique identifiers (CUIs). We participated in Task 1 and developed a clinical disorder recognition and encoding system. The proposed system consists of two components: a machine learning-based approach to recognize disorder entities and a vector space model-based method to encode disorders to UMLS CUIs. The challenge organizers manually annotated disorder entities and corresponding UMLS CUIs in 298 clinical notes, of which 199 notes were used for training and 99 were for testing. Evaluation on the test data set showed that our system achieved the best F-measure of 0.750 for entity recognition (ranked first) and the highest F-measure of 0.514 for UMLS CUI encoding (ranked third), indicating the promise of the proposed approaches.
Clinical natural language processing (NLP) has received great attention in recent
years because it is critical to unlock information embedded in clinical documents in
the secondary use of electronic health records (EHRs) data for clinical and
translational research. Clinical concept extraction, which recognizes clinically relevant
entities (e.g., diseases, drugs, labs etc.) in text and maps them to identifiers in standard
vocabularies (e.g., Concept Unique Identifier (CUI) defined in Unified Medical
Lan* corresponding author
guage System (UMLS) [
In 2013, the ShARe/CLEF eHealth Evaluation Lab (SHEL) organized three shared tasks on natural language processing (NLP) and information retrieval (IR): 1) clinical disorder extraction and encoding to Systematized Nomenclature Of Medicine Clinical Terms (SNOMED-CT), 2) acronym/abbreviation identification, and 3) retrieval of web pages based on queries generated when reading the clinical reports. The Task 1 on clinical disorder extraction is similar to the 2010 i2b2 challenge on clinical problem extraction. However, there are two major differences between these two tasks: 1) ShARe/CLEF task allowed disjoint entities, while 2010 i2b2 clinical problem extraction only dealt with entities of consecutive words; and 2) ShARe/CLEF task required mapping disorder entities to SNOMED-CT (using UMLS CUIs), which was not required in the 2010 i2b2 challenge.
In this paper, we describe our system for Task 1 of the 2013 ShARe/CLEF challenge. Our system consists of a machine learning based approach for disorder entity recognition and a Vector Space Model (VSM) based method for mapping extracted entities to SNOMED-CT codes. Evaluation by the organizers showed our system was top-ranked among all participating teams. 2
Fig. 1 shows the overview architecture of our systems for the first task of the ShARe/CLEF eHealth 2013 shared task. It is an end-to-end system of two components: disorder entity recognition and encoding. The first component consists of five modules. As the clinical narrative supplied by the organizer was not well formatted, we developed rule-based modules to detect the boundary of sentences and tokenize them for each note at first, and aligned the preprocessed note back to the original one at last. The other components were presented in the following sections in detailed.
Fig. 1. The overview architecture of our disorder concept extraction systems for the first task of the ShARe/CLEF eHealth 2013 shared task. 2.1
The organizers collected 298 notes from different clinical encounters including radiology reports, discharge summaries, and ECG/ECHO reports. For each note, disorder entities were annotated based on a pre-defined guideline and then mapped to SNOMED-CT concepts represented by UMLS CUIs. If a disorder entity cannot be found in SNOMED-CT, it will be marked as “CUI-less”. The data set was divided into two parts: a training set of 199 notes that were used for system development, and a test set of 99 notes for evaluating systems. In the training set, 5811 disorder entities were annotated and mapped to 1007 unique CUIs or CUI-less. The test set contained 5340 disorder entities with 795 CUIs or CUI-less. Table 1 shows the counts of entities and CUIs in the training and test datasets. 2.2
In machine learning-based named entity recognition (NER) systems, annotated data are typically converted into a BIO format, where each word is assigned into one of three labels: B means beginning of an entity, I means inside an entity, and O means outside of an entity. Thus the NER problem is converted into a classification problem to assign one of the three labels to each word. As mentioned previously, one challenge of this task is that some disorder mentions (>10%) were disjoint, which could not be directly solved using the traditional BIO approach, which only works on entities with consecutive words. Therefore we developed different strategies for consecutive entities and disjoint entities. For consecutive disorder entities, we labeled words Dataset Training Test using traditional BIO tags. For disjoint entities, we created two additional sets of tags: 1) D{B, I} was used to label disjoint entity words that are not shared by multiple concepts (called non-head entity); and 2) H{B, I} was used to label head words that belonged to more than two disjoint concepts (called head entity). Figure 2 shows some examples of labeling consecutive and disjoint disorder entities using our new tagging sets. In this approach, we need to assign one of the seven labels {B, I, O, DB, DI, HB, HI} to each word. When converting labeled words to entities, we defined a few simple rules. For example, one rule for head words is “for each disjoint head entity, combine it with all other non-head entities to form final disorder entities”.
Sentence 1: “The left atrium is dilated .” Encoding: “The/O left/DB atrium/DI is/O dilated/DB ./O” Sentence 2: “The aortic root and ascending aorta are moderately dilated .” Encoding: “The/O aortic/DB root/DI and/O ascending/DB aorta/DI are/O moderately/O dilated/HB ./O”
We investigated two machine learning algorithms for disorder entity recognition. One is Conditional Random Fields (CRFs), which is a representative sequence labeling algorithm and is suitable for the NER problem. Another one is Structural Support Vector Machines (SSVMs), which was proposed by Tsochantaridis et al. [23] in 2005 for structural data, such as trees and sequences. It is an SVMs-based discriminative algorithm for structural prediction. Therefore, SSVMs combines the advantages of both CRFs and SVMs and is suitable for sequence labeling problems as well. CRFsuite (http://www.chokkan.org/software/crfsuite/) and SVMhmm (http://www.cs.cornell.edu/people/tj/svm_light/svm_hmm.html) were used as implements of CRF and SSVM respectively.
For features, we used bag-of-word, part-of-speech (POS) from Stanford tagger
(http://www-nlp.stanford.edu/software/tagger.shtml), type of notes, section
information, word representation from Brown clustering [
We treated disorder entity encoding as a ranking problem, where each recognized disorder entity was considered as a query and candidates terms in UMLS as documents. The Vector Space Model (VSM) was used in this task. The process consists of two steps: 1) generate candidate CUIs from UMLS; and 2) rank candidate CUIs and then take the top ranked CUI as the system’s output. We applied following criteria to select candidate CUIs from UMLS for a given disorder entity: the corresponding terms of a candidate CUI should contain all words in the disorder entity (except stop words). For each candidate CUI, a vector containing its words, weighted by term frequency–inverse document frequency (tf-idf) derived from entire UMLS/SNOMED-CT terms, was created. The cosine similarity between a disorder entity vector and a candidate CUI vector was calculated and used to rank candidate CUIs. The top ranked CUI was then selected as the correct CUI of the entity. In order to leverage the training data, we further built a limited VSM-based encoding system by using CUIs/terms and entities occurred in the training set only, instead of the entire UMLS. When processing the test set, we first determined whether an entity occurred in the training set or not. If it did, we used the limited VSM-based encoding system to predict the corresponding CUI. Otherwise, we used the general VSM-based encoding system that was built on entire UMLS. 2.4
Our system was developed and trained using the training set (199 notes) and was
evaluated using the test set (99 notes). All parameters of CRF and SSVM were
optimized by 10-fold cross-validation on the training dataset. The performance of
disorder entity recognition were evaluated by precision, recall and F-measure in both
“strict” and “relaxed” modes, where “strict” refers that a concept is correctly
recognized if and only if the starting and ending offsets of it is exactly same as a disorder
mention in the gold standard, and “relaxed” refers that a disorder mention is correctly
recognized as long as it overlaps with any disorder mention in the gold standard. For
encoding of SNOMED-CT, all participating systems were evaluated using accuracy
only, in “strict” and “relaxed” modes, as defined in [
F-measures of 0.750 under “strict” criterion and 0.873 under “relaxed” criterion,
ranked first in the challenge. For SNOMED encoding, our system achieved the best
accuracy of 0.514, ranked third in the challenge.
Although a number of existing clinical NLP systems such as MedLEE [
In this study, we developed a clinical disorder recognition and encoding system
that combines a machine learning based approach for entity recognition and a
VSMbased approach for UMLS concept mapping. Our system was top-ranked among all
participating teams, indicating the promise of proposed approaches. However, there is
still much room for further improvement. First, our proposed method for disjoint
entity recognition has limitations. For example, if a sentence has multiple disjoint entities,
our current simple rule-based strategies would not be able to resolve the ambiguity
and will produce wrong combinations of disorder entities as shown in Fig 3, where
there are two disorder entities in the given sentence: “blood … on his tongue” and
“pupils … pinpoint”, which are represented by “blood/DB … on/DB his/DI
tongue/DI” and “pupils/DB … pinpoint/DB” respectively, but parsed into one
disorder entity “blood … on his tongue … pupils … pinpoint” by our strategies. Thus,
more sophisticated methods for disjoint concept recognition should be investigated in
future. In addition, our VSM-based method to map entities to UMLS CUIs is not
optimal. When compared with the top ranked team on UMLS CUI mapping, our
system achieved better performance on entity recognition, but lower accuracy on CUI
mapping, indicating the weakness of our encoding step. A few possible aspects for
further improvement are: 1) use other types of information as features for building
vectors, such as context, type of notes, section information and so on; 2) explore other
ranking algorithms such as Support Vector Machines [
This study is supported in part by grants from NLM R01LM010681, the Office of the National Coordinator for Health Information Technology 10510592, NCI 1R01CA141307, and NIGMS 1R01GM102282. We also thank the ShARe/CLEF eHealth shared task 2013 organizers, who were funded by the United States National Institutes of Health with grant (R01GM090187).