This paper describes a system for span detection and normalization of disorder mentions in clinical notes as de ned in Tasks 1a and 1b of the 2013 ShARe/CLEF eHealth Evaluation Lab [1]. We take a supervised learning, chunking-based approach to identifying disorder spans. In particular, our system introduces a method for identifying the spans of disjoint and overlapping disorder mentions using relation extraction and semantic role labeling (SRL). Our primary objective was to demonstrate the utility of relations for resolving the spans of disjoint disorder mentions. We used a CRF-based sequence labeler to extract initial disorder spans. Using these spans, we applied a locational relation extractor and SRL system to locate pairs of spans belonging to the same disorder mention. We used a dictionary-based approach to disorder normalization. Under strict evaluation for Task 1a, our system performed 3rd out of the 15 best performing systems for each team, achieving an F-measure of 0.687. For Task 1b, our system achieved a strict F-measure of 0.441. Our disjoint span resolution system signi cantly improved the performance of our system in both tasks, achieving a 5.5% increase over our baseline system in Task 1a and a 7.8% increase in Task 1b.
Task 1 of the 2013 ShARe/CLEF eHealth Evaluation Lab [
We approach the task of identifying disorder spans as a supervised sequence
labeling problem. Our system uses relation extraction and semantic role labeling
to identify the spans of disjoint and overlapping disorder mentions. Our baseline
system for the disorder span identi cation task uses a CRF (conditional random
eld) sequence labeler. The clinical Text Analysis and Knowledge Extraction
System (cTAKES) is used to preprocess the data [
Before training the sequence labeler, we split up multi-span annotations into individual disorder annotations. For example, right atrium and dilated are annotated as disjoint spans within the disorder annotation right atrium dilated in the sentence \The right atrium is moderately dilated." These disjoint spans are automatically annotated as independent disorder spans, while the original annotation is removed. 2.2
The majority of preprocessing is accomplished using cTAKES components. Our baseline system's preprocessing pipeline consists of tokenization, sentence segmentation, part of speech tagging and NP-chunking. Finally, the cTAKES dictionary lookup module is applied over NPs found during chunking, providing entity mention annotations (drugs, diseases/disorders, signs/symptoms, anatomical sites, labs, procedures, and their associated CUIs). 2.3
Our system uses the CRFsuite linear-chain CRF implementation for sequence
labeling [
Our system uses a combination of orthographic/lexical, syntactic and semantic features commonly used in named entity recognition. All orthographic/lexical and syntactic features are repeated in a window around the current token. A window of +/-2 tokens was found to provide the best performance on the training data. The normalized form of each token is extracted using a hand-crafted dictionary and is then used as a feature.
Raw Clinical Text cTAKES Pre-Processing ClearTK Feature Extraction CRFsuite Sequence Labeling
Relation Extraction Disjoint Span Resolution
CUI Normalization
We also incorporate two domain-dependent features. Discourse section refers to the heading preceding the current section of text. With the observation that sections were generally colon-delimited in the task data, we naively identify the current discourse section as the text immediately preceding the previous colon. Document type is extracted from the lename of the document currently being processed. In the task data, the document type (ECHO REPORT, ECG REPORT, DISCHARGE SUMMARY, or RADIOLOGY REPORT) could be found in the lename directly before the le extension (.txt).
Entity mentions extracted using the cTAKES the dictionary lookup module are used in the system as semantic features. Each entity type overlapping the span of the current token is included as a feature. 2.4
After identifying an initial set of disorder spans during sequence labeling, we apply our disjoint span resolution system to connect pairs of disjoint spans belonging to the same entities. This is accomplished using relations extracted with the ClearNLP SRL module and the cTAKES relation extractor. Where locational relations extracted from cTAKES or predicate-argument relations extracted using SRL are found, we combine the disjoint spans corresponding to their constituents into single entities.
To identify locational relations, we use the cTAKES relation extractor
module [
Finally, if a relation's constituent elements are contained within disorder spans from the initial set found during sequence labeling, we create a new, multispan disorder mention from the corresponding spans. The old disorder spans are discarded. 2.5
To assign CUIs to the identi ed disorder spans, we apply a simple rule-based system in conjunction with the cTAKES dictionary lookup algorithm. We query cTAKES using the contents of each disorder mention as a lookup window. Because queries typically generate multiple results, we take several steps to lter candidate CUIs. First, we only consider identi ed annotations that cover the entire span. For example, e usion and pericardial in pericardial e usion are removed from consideration. Secondly, we only consider candidates that have TUIs (semantic type unique identi ers) belonging in the Disorder/Finding semantic groups. Finally, if no candidates are found, we mark the disorder span as CUI-less.
Two runs were submitted for each task. In both tasks, CLEAR NoRel corresponds to our baseline system without any disjoint span resolution while CLEAR Rel incorporates our disjoint span resolution system. The tables below summarize the results for Task 1a and 1b using the SNOMED-CT 2011 gold standard. Also included in the tables are the results for the three best-performing systems out of all the teams excluding our system.
In Task 1a with strictly matching spans, CLEAR Rel performed 3rd out of the 15 best performing systems for each team. Adding the disjoint span resolution system increased our baseline system's F-score by 0.036, a 5.5% improvement. For CUI normalization with strict evaluation, our system observed a 0.032 increase in F-score after adding disjoint span resolution, a 7.8% improvement. We've introduced a system for disorder span detection and CUI normalization. An initial set of disorder spans are identi ed using a CRF-based IOB sequence labeler. CUI normalization is accomplished using the cTAKES dictionary lookup module and several simple rules. Using relations to resolve disjoint and overlapping spans signi cantly improves system performance in both disorder span detection and CUI normalization.
A more sophisticated system for normalizing abbreviations and acronyms
would likely improve performance and make the system extensible to domains
beyond ECGs, echo reports, radiographs and discharge summaries. Furthermore,
CRFs have no built in capacity for capturing long distance dependencies [
Although the cTAKES relation extractor and ClearNLP SRL system correctly identi ed many relations corresponding to entities with disjoint spans, many were also missed. For example, neither locational relations nor predicateargument relations capture disjoint spans in coordination structures such as \right and left ventricles enlarged" in which right ventricles enlarged and left ventricles enlarged are both disjoint entities. Better results might be achieved by training a model to identify the relations that correspond to disjoint spans using the output from a simple dependency parse.
Acknowledgments. Many thanks to Martha Palmer, James Martin, Steven Bethard, Guergana Savova, and Sameer Pradhan for providing a wealth of ideas and advice in this task. This work was supported by award number 90TR002 provided by the ONC's Strategic Health IT Advanced Research Projects (SHARP). We gratefully acknowledge the e orts of all of the task organizers as well as the Shared Annotated Resources (ShARe) project funded by the United States National Institutes of Health with grant number R01GM090187.