In this paper we present two systems that address the issues of disorder recognition and normalization submitted by the authors as de ned by the CLEF/ShARe Evaluation Lab. The rst approach to the tasks formed a baseline approach using the cTakes system. Our second approach leveraged Structural Support Vector Machines with an array of feature types including lexical, semantic and cluster based knowledge. The recognition di ers from typical NER tasks in that disorder spans may be disjoint i.e. a disorder can be non-contiguous. To address this issue we introduce a new tag type to annotate tokens occurring between disorders.
The CLEF/ShARe Evaluation Lab [1] provided a platform for comparative evaluation of clinical NLP about information retrieval technologies. The evaluation lab was comprised of three challenges:
2. Abbreviation recognition & normalization 3. Information Retrieval
For the purposes of this paper, we focus on the authors' submissions to the task of disorder recognition and normalization.
Our submissions had two goals: the rst submission is based on the cTakes processing system, allowing us to gauge the suitability and performance of readyto-use systems for the described tasks. The second submission is a machinelearning system built by the authors. This system allows for the identi cation of what feature sets aid the recognition task.
In the next section we present a brief overview of the task at the CLEF/ShARe Evaluation Lab.
In this Section we present an overview of Task 1 at the ShARe/CLEF eHealth Evaluation Lab in which the authors participated. The dataset for this task comprised of 300 clinical reports (Discharge Summary, Radiology Report, Echo report) with disorder spans and normalized concepts from the UMLS Metathesaurus annotated. The training data comprised of 200 reports, with the released test data containing 100 reports.
Task 1 is divided into two subtasks: (a) disorder recognition, (b) disorder normalization. A Disorder is de ned to an entity that may occur under the allowed groups in the UMLS Metathesaurus, as shown in Table 1. Though primarily a Named Entity Recognition (NER) task, it di ers with previous clinical NER challenges, such as the i2b2 challenge. Firstly, an entity may be disjoint. That is to say that a recognized entity may or may not be a contiguous sequence of tokens, as is often a requirement in named entity challenges. For example, in Sentence 1., we see the concept text Epstein's anomaly separated by the token sequence cardiac valve.
The second unique aspect to the task requires the normalization of
identied concept spans. For example, non-standard terminology may be used, or the
concept is interrupted by a span of text, issues that may pose problems to
information retrieval techniques. To perform normalization, spans are rst identi ed.
Following identi cation, spans are then mapped to ontology concepts, in the
case of this challenge the UMLS metathesaurus is used. For the identi ed
concept in sentence (
This section describes the two systems submitted to the CLEF/ShARe Task 1 in disorder recognition and normalization. 3.1
Prior to recognizing disorders, the corpus must rst be preprocessed. Firstly, documents were split into sentences using LingPipe1 tools. Following sentencesplitting, the corpus is then passed through the cTakes system [2]. cTakes performs syntactic and semantic processing of the dataset as well as the recognition of named entities. The design of systems using this information is discussed in the following sections. 3.2
cTakes facilitates information extraction from electronic medical health records. It is a comprehensive toolkit for processing clinical text, including abilities to detect named entities and map entities to CUI's in the UMLS metathesaurus. However, there are two issues that make cTakes an unsuitable candidate for disorder recognition and normalization. Firstly, cTakes cannot recognise disjoint entities. Secondly, as cTakes maps to all CUI's it raises issues in that we only require disorder normalization.
To achieve our aims, we apply simple post-processing rules on cTakes output in order to retrieve the required entities. Firstly, all recognised entities are ltered according to the allowed semantic groups that represent disorders in the UMLS Metathesaurus. In order to recognize disjoint disorders, our system performs a check that if another disorder with a matching CUI occurs within 10 tokens, those disorders are linked to create a disjoint entity. 3.3
Despite the popularity of CRF's and other Markov approaches, Structural SVM's have been shown to achieve state-of-the-art performance with less training time on clinical datasets [3]. For this reason, the authors have designed an approach that leverages Structural SVM's as shown in Figure 1. Traditionally, tagging tasks such as Named Entity Recognition (NER) use the BIO (B-beginning, Iintermediate, O-outside) format. However, pre-submission experiments by the authors replicated results in the literature showing that improvements can be achieved using the BIESO (B-beginning, I-intermediate, E-end, S-single token concept, O-outside), particularly on long and short concepts. A key di erence in the disorder recognition task and previous NER tasks is the allowance of disjoint concepts whereby disorders are not necessarily contiguous sequences of disorder tokens. To address this issue, the authors' use a modi ed BIESTO (T-beTween) tagging format that allows the tagging of tokens that occur between members of a disorder span as shown in Figure 2 COINLPLEUCTTION
Splitter cTakes
Feature Extraction
SSVM CUI Extractor
SPANS CUIS
Rules
NoINrmPUalTized Disorders
Features used in our machine-learning based system leveraged several types of information discovered in the text using the cTakes system and rules developed by the authors. Feature types included lexical, syntactic, semantic and clustering based features. To account for disjoint spans several features were introduced. Firstly, checks are performed to identify if matching CUI's occur in the same sentence. Features also analyze syntactic and part-of-speech information such as when the current token is a preposition to provide clues in identifying tokens that occur between spans. The normalization system is the same as described in Section 3.2 with the exception that the spans input to the system are those identi ed in the machine-learning system, rather than the cTakes system. 4
In this Section we present the evaluation metrics and results achieved by the systems described in the previous section. 4.1
The disorder recognition systems are evaluated by the metrics precision (P), recall (R) and f-score (F):
P =
T P
T P + F P
(
R =
T P
T P + F N F = 2
P recision Recall
P recision + Recall where TP is the number of true positives, FP is the number of false positives, FN is the number of false negatives.
For the task of recognising the spans of disorders in text, exact and inexact calculations of the above metrics are used. For an exact match, the begin and end o sets must match exactly. For an inexact calculation a candidate o set is counted as a True Positive if its spans overlap with the span of a gold standard annotation.
Accuracy is used as the evaluation metric for the normalization task. It is de ned as follows:
Accuracy =
CORRECT
T OT AL where CORRECT is the number of disorders with correct span and CUI, TOTAL = Total number of disorders
Similar to the recognition task, there is both a strict and relaxed metric. For the strict metric, the gold-standard annotation is used as the count for Total number of disorders. In the relaxed metric, the system is evaluated with respect to only annotations detected by the system. In the next Section, we present the results of the systems submitted by the authors. 4.2
This section provides the results of the authors submissions to the CLEF/ShARe Evaluation Lab. Our machine learning based approach to disorder recognition, UCDCSI.1, achieved competitive results across exact and inexact evaluations. The baseline cTakes system UCDCSI.2 proved unsuccessful in recognizing disorder spans. Though not the worst performing system, it appears that a custom-built system for disjoint disorder recognition is required.
Examining the performance of the system on the exact metric showed some
recurring errors. Firstly, given the strict nature of the metric, partial matches
were penalised heavily. For example, the system recognised the instance
pulmonary hypertension instead of the gold standard annotation primary pulmonary
hypertension. However, issues such as this are addressed by the inexact metric.
(
System Name P R F
UCDCSI.1 0.922 0.758 0.832
UCDCSI.2 0.512 0.339 0.408
Table 3. Disorder Recognition Inexact-Spans A second key cause of false positives would be the combination of a body-part and a modi er, such as normalized gallbladder.
This issue also extended to tests and treatments relating to a body part, such as Liver function tests. The nal group of false positives were often negated disorders or those that featured some other assertion status, such as conditional or hypothetical. However, assertion status also generated issues with false negatives leading this to be a topic for further investigation. Typically, acronyms and abbreviations were a source of false negatives. While the system performed well in recognizing disjoint disorders, the performance weakened in detecting long disorders e.g. mitral valve prolapse since adolescence who developed signi cant regurgitation due to the long spaces between disjoint entities being atypical in the training set.
Both UCDCSI.1 and UCDCSI.2 relied on cTakes and post-processing rules to normalize concepts to CUI's in the UMLS Metathesaurus. However, this approach was far from e ective, achieving moderate performance on the relaxed accuracy metric. This posits the idea that a free-standing module is required in order to correctly map identi ed concepts to CUI's. Analysis of the system's performance show that while cTakes may correctly map the CUI, its output produces several CUI's. Therefore, a more elegant set of post-processing rules may see the normalization process improve. For example, similarity measures may be used between an identi ed span and the textual representation of the identi ed CUI. 5
In this paper, we have described a machine-learning based disorder recognition system using Structural SVM's and a novel BIESTO based tagging approach that facilitates the detection of disjoint entities. The system posted competitive results providing a solid foundation for future work. In particular, future work will focus on the normalization of recognised disorders. The approaches in this paper use frequency counts to map concept identi ers to disorders, future work may use semantic similarity measures to correctly identify normalized concepts.
This work was made possible through the Shared Annotated Resources (ShARe) project funded by the United States National Institutes of Health with grant number R01GM090187. We also wish to acknowledge the support of Science Foundation Ireland, who fund this research under grant number 10/RFP/CMS2836.