This paper describes the THCIB systems that used in the ShARe/CLEF eHealth 2013 task 1. We implemented two baseline systems and a combination system using the existing technologies. One baseline system is built using MetaMap. We built another baseline system using cTAKES. Furthermore, we developed the combination system with a system combination method. The results of combination system were submitted because the combined results performed better than either single system. We also report the experimental results on the training set and the test set.
The ShARe/CLEF eHealth Lab 2013 task 1 aims at named entity recognition and
normalization of disorders [
For the time limitation, we built the baseline systems and combination system using existing technologies. The results of combination system were submitted due to better performance. In this paper we describe the workflow of the baseline systems and combination system. And we also present the experimental results on the training set and the test set.
The reminder of this paper is structured as follows. In section 2, we present an overview of our baseline systems. In section3, we describe the system combination method. The experiments and analysis of the result are described in section 4. We give the conclusion in section 5.
Clinical
Text Clinical
Text
We built two baseline systems for task 1. Both baseline systems are implemented
using open source software (OSS). One is built using MetaMap [
MetaMap is a highly configurable program developed by the National Library of
Medicine (NLM) to map biomedical text to the UMLS (Unified Medical Language
System) [
The flowchart of baseline system 1 is shown in Fig. 1. In the baseline system 1, the clinical text is processed as following steps: 1) the clinical text is sent to MetaMap; 2) the MetaMap processes the clinical text and maps all Metathesaurus concepts in the clinical text to UMLS. The concepts will be saved in an XML file. 3) Post-processing the XML file, and extract the disorders and the corresponding CUIs; 4) output the disorders and CUIs.
MetaMap
Post-processing
cTAKES (Apache clinical Text Analysis and Knowledge Extraction System) is an
open source natural language processing system for information extraction from
electronic medical record clinical free-text [
The flowchart of baseline system 2 is shown in Fig. 2. The clinical text is processed as following steps: 1) the clinical text is sent to cTAKES; 2) the cTAKES processes the clinical text, and extract named entities. The extracted named entities will be stored in an XCAS file; 3) post-processing the XCAS file, and extract the named entities which belong to disorders and the corresponding CUIs; 4) output the disorders and CUIs.
cTAKES
Post-processing Fig. 2. Flowchart of baseline system 2.
Disorders & CUIs
In order to take advantage of the strengths of two baseline systems, we employ a
simple but efficient system combination approach to combine the results of the
baseline systems [
ALGORITHM 1: Set Z empty; // for (i=1; i<=m; i++) { // Initialization
add xi to Z; // } for (j=1; j<=n; j++) { // Combination if yj conflict with Z:
discard yj; else:
add yj to Z; }
This is because that the baseline system 1 has higher precision, while the baseline system 2 has higher recall. 4
4.1
The training set contains 200 clinical reports, and totally 5874 disorders. We used all of the training set to evaluate the performance of each system. The test set contains 100 clinical reports. We will give evaluation results on both training set and test set. 4.2
Precision, recall and F1 measure are used in this evaluation. Two conditions are setup. One is strict, which means that the recognized words are perfectly matched; the other is relaxed, which means that the recognized words have overlap with the gold standard. 4.3
We evaluate three systems using training set. The results of task 1a and task 1b are shown in Table 1 and Table 2, respectively. The baseline1 is the results of baseline system 1; the baseline2 is the results of baseline system 2; and the combination is the results of the combination system.
From Table 1, we can find that the baseline1 performs better in the strict metric while baseline2 performs better in the relaxed metric. According to analysis of the results, we find that the average length of baseline2 results is shorter than baseline1 results, but the quantity is larger than the baseline1 results. This leads to a higher recall but lower precision.
From the results of combination, we find that the recall get great improvement. And the F-score can also be improved, especially for the relaxed metric.
Table 2 shows the evaluation results of task 1b. The baseline1performs better in the strict metric and baseline2 performs better in the relaxed metric. After combining the results of two baseline system, the results can improve from 0.401 to 0.455 in the strict metric and acceptable decrease in the relaxed metric.
For the time limitation, our main purpose is using the existing technologies to build baseline system for disorder recognition and verify the performance of the existing technologies. We built two baseline systems using OSS for ShARe/CLEF eHealth task 1. And we also built a combination system by combine the results of two baseline systems. The evaluation results on the training set and the test set show that the combination system can perform better than single baseline system.
This research is supported by Canon Inc. (No. QIM2013). The Shared Annotated Resources (ShARe) project is funded by the United States National Institutes of Health with grant number R01GM090187. We also appreciate the valuable comments from the task organizer.