This paper describes the THCIB systems that used in the ShARe/CLEF eHealth Lab 2013 task 2. We built a baseline system using open source software, and improve the performance by adding dictionaries. The dictionary is built from training set and web resource using the existing technologies. The experimental results show that adding dictionary of acronym/abbreviation can improve the performance significantly.
The ShARe/CLEF eHealth Lab 2013 task 2 aims to normalize of
acronyms/abbreviations (AAs) [
In this paper we describe the baseline system and the dictionaries we used to improve the performance. And we also describe the experimental results on the training set and the test set.
The reminder of this paper is structured as follows. In section 2, we present an overview of our baseline system. In section3, we describe how to build the dictionaries. The experiments and analysis of the results are described in section 4. We give the conclusion in section 5.
The baseline system for task 2 is implemented using the cTAKES [3]. The cTAKES (Apache clinical Text Analysis and Knowledge Extraction System) is an open source natural language processing system for information extraction from electronic medical record clinical free-text. It can process the clinical text and identify the clinical named entities from various resources including the UMLS [4].
The flowchart of baseline system is shown in Fig. 1.
Clinical
Text cTAKES
Post-processing
CUIs
In the baseline system, the clinical text is processed in following steps: 1) the clinical text is sent to cTAKES; 2) the cTAKES processes the clinical text and maps all concepts in the clinical text to UMLS. The concepts will be saved in an XCAS file. 3) Post-processing the XCAS file, and extract the CUI of each AA which has been annotated. If an AA has no corresponding CUI, a “CUI-less” tag will be given; 4) output the CUIs. 3
The baseline system has very low performance. Experiment on the training set shows that about 80% AAs can’t be recognized correctly. An intuitive method to improve the AA recognition performance is adding dictionary as external resource [5]. We build two kinds of dictionaries for task 2. 3.1
The CUI dictionary is a dictionary which maps the AA to the corresponding CUI directly. The CUI dictionary is built using the training set. There are 3660 AAs in the training set. After combination, the CUI dictionary contains 668 entries [5]. An example of CUI dictionary entry is “BP” “C0005823”. 3.2
The full name dictionary maps the AA to its full name. This dictionary is built using two resources. One is the training set, we extract 668 entries. The other is a web medical dictionary which contains 2180 entries [6]. After the combination, the full name dictionary has 2725 entries. An example of full name dictionary entry is “BP” “blood pressure”.
One AA may map to more than one full name. For example, the AA “HA” may map to “headache” or “herpangina” in different context. For this case, we rank the full name using the co-occurrence of the AA and its full name in a large scale web page corpus which contains more than 1.6 million web pages [7,8]. The full name which has the most co-occurrence with the AA will be reserved. 3.3
As the baseline system, the input is a sentence with annotation of AA, and the output is the corresponding CUI. We use a dictionary lookup method [4]. The system works as follows:
1) Look up the AA in the CUI dictionary. If find the AA in the dictionary, output the corresponding CUI;
2) If can’t find the AA in the CUI dictionary, look up the AA in the full name dictionary;
3) If find the full name of the AA, send the full name to the cTAKES; else, send the whole sentence to the cTAKES;
4) Extract the CUI of the AA. If the AA has no CUI, give it a “CUI-less” tag. 4
4.1
The training set contains 200 clinical reports, and totally 3660 AAs. We used all of the training set to build the dictionary and evaluate the performance of the system. The test set contains 100 clinical reports. We will give the evaluation results on training set and test set. 4.2
Precision is used in this evaluation. Two conditions are setup. One is strict, which means that the recognized words are perfectly matched; the other is relaxed, which means that the recognized words have overlap with the gold standard. 4.3
We use all the training set to evaluate the baseline system. The evaluation results are shown in Table 1.
From Table 1, we can find the baseline system has very low performance on the AA normalization.
Because the dictionary is built using the training set, we use a five-fold cross validation to verify the CUI dictionary. The results are shown in Table 2.
From Table 3, we can find that only adding full name dictionary can improve the accuracy from 0.208 to 0.409. And only adding CUI dictionary can improve the accuracy to 0.885. The accuracy of adding two dictionaries is slightly lower than only using the CUI dictionary. This is caused by the full name dictionary because one full name may map to several CUIs. And current system doesn’t consider the disambiguation problem. Though using full name dictionary will reduce the accuracy slightly, the system will be more robust because it can cover more AAs. So we select this system to process the test set. The official results are shown in Table 4.
From Table 4, we can find that the performance on the test set is similar to the performance on the training set. This means that the acronym/abbreviation dictionaries work well on the test set. 5
For the time limitation, our purpose is using the existing technologies to build the baseline system for acronym/abbreviation normalization and verify the performance of the existing technologies. We built a baseline system using OSS for ShARe/CLEF eHealth task 2. In order to improve the performance, we built two kinds of dictionaries as the external resource. One is the CUI dictionary which maps the AA to CUI directly; the other is the full name dictionary which maps the AA to its full name. The CUI dictionary is built using the training set, and the full name dictionary is built using the training set and the web resource. The experimental results show that adding dictionary to the baseline system can improve the performance significantly.
This research is supported by Canon Inc. (No. QIM2013). The Shared Annotated Resources (ShARe) project is funded by the United States National Institutes of Health with grant number R01GM090187. We also appreciate the valuable comments from the task organizer. 2. Unified Medical Language System (UMLS), http://www.nlm.nih.gov/research/umls/ 3. Apache cTAKES, http://ctakes.apache.org/index.html 4. Savova, G., Masanz, J.J., Ogren, P., Zheng, J., Sohn, S., Kipper-Schuler, K., Chute, C.: Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component, evaluation and applications. J Am Med Inform Assoc. 17, 507-513(2010) 5. Borthwick, A., Sterling, J., Agichtein, E., Grishman, R.: NYU: Description of the MENE
Named Entity System as Used in MUC-7. In proc. of MUC 7, 1998. 6. Medical abbreviations, http://www.abbreviations.com/acronyms/MEDICAL 7. Genetics Home Reference, http://ghr.nlm.nih.gov/ 8. Diagnosia, http://www.diagnosia.com/en/