Named entity recognition of biomedical text is the shared task 1b of the 2015 CLEF eHealth evaluation lab, which focuses on making biomedical text easier to understand for patients and clinical workers. In this paper, we propose a novel method to recognize clinical entities based on conditional random fields (CRF). The biomedical texts are split into sections and paragraphs. Then the NLP tools are used for POS tagging and parsing, and four groups of features are extracted to train the entity recognition model. In the subsequent phase for entity normalization, the MetaMap of Unified Medical Language System (UMLS) tool is used to search for concept unique identifiers (CUIs) category. In addition, CRF++ package is adopted to recognize clinical entities in another phase for entity recognition. The experiments show that our system named as WI-ENRE, is effective in the named entity recognition of biomedical texts. The Fmeasure of EMEA and MEDLINE reach to 0.56 and 0.45 respectively in exact match.
With the application of EMRs, hospitals and medical institutions generate masses of biomedical text. Based on biomedical text, the medical big data analytics and the building of heath knowledge network are the critical problem. As a precondition to solve the problem, named entity recognition can provide a solution to extract information and knowledge from biomedical text. Hence, the named entity recognition is becoming a research hotspot.
Biomedical text contains a wealth of information on patients covering their hospital
stays, including health conditions, diagnoses, performed tests and treatments. Named
entity recognition form biomedical text has a good research foundation[
Methods for entity recognition can be roughly divided into three categories: rulebased, machine learning methods and a combination of both. The method of rule-based mainly relies on proper nouns dictionaries and rules which wrote by language experts or domain experts to identify the clinical entities. Compared to rule-based methods, many more researchers choose machine learning methods on entity recognition.
In this paper, we propose a novel method for task 1b of CLEFeHealth 2015. In order to testify this method, we design a named entity recognition system, WI-ENRE, which adopts machine learning method based on conditional random fields for the nine categories and lexicon-based approach for geographic areas.
The rest of this paper is arranged as follows. In Sec. 2, we discuss the materials and methods in detail, and also focus on feature optimizing selection. Moreover, we conduct the experiments to testify the effectiveness of WI-ENRE in Sec. 3. In Sec. 4, we conclude this paper and discuss the directions for further work. 2
In this study, the dataset which is called QUAERO French Medical Corpus[
In the process of entity recognition and entity normalization, some related resources are used, which contain Stanford Parser based on French and UMLS tool. Then, the feature selection will be described as the significant part in this paper. Finally, the principle of conditional random field algorithm will be detailed in Sec. 2.4. 2.1
The corpus is provided by the 2015 CLEFeHealth evaluation lab. The task 1b consists of clinical named entity recognition and entity normalization from the file of MEDLINE titles and EMEA documents.
In order to testify the method of entity recognition, the training set provided by CLEFeHealth is divided into two parts: the dataset for training which contains 676 documents and a total of 22,160 words, and the testing set contains 168 documents and a total of 3,336 words. Moreover, the number of entity and deduplicated entity are counted, respectively (as shown Tab. 1). In Tab. 2, we also give a few statistics for each category in the training corpus. 2.2
Stanford Parser. As an existing open source toolkit, Stanford Parser is utilized to split sentences of the biomedical text. Furthermore, Stanford Parser also provides the function of POS tagging for multi-languages, such as English, Chinese, French, German and so on.
UMLS. Unified Medical Language System (UMLS) is used for mapping clinical entity
to the unique concept identifiers (CUIs). And MetaMap[
Before model training, a large number of features need to be extracted from biomedical texts. The features can be categorized into four groups: lexical features, orthographic features, context features and lexicon features, listed in Tab. 3.
Lexical features use the first and the last four characters of token to identify the categories of entities. The POS of a token is helpful in named entity recognition. The Stanford Parser tool is used to get POS tag of token, which is learnt on open domain corpus and supports multiple languages by loading template.
The tokens similar in shape can help the classifier “memorize” whether the token belong to one type of the entities. We replaced uppercase letters, lowercase letters, letters with diacritics and digits in a token by “A”, “a”, “b” and “0”, respectively. Length of a token is a significant feature to clinical entity recognition. Similarly, information of capital letters is also a strong feature to help us identify the entities which always consist of uppercase letters. For example, the tokens of “Bio-safety Cabinet”, “CT” and other proper noun can be identified by capital feature.
The context features of the classifier contain the lowercase, first four characters, last four characters, POS tags of two tokens before and after the current token.
lowercase of the current token first four characters of the current token last four characters of the current token POS of the current token shape of the current token length of the current token whether the current token contains a letter whether the current token begins with a capital letter whether all characters in the current token are capital letters whether the current token contains a digit whether all characters in the current token are digits whether the current token consists of letters and digits first four characters of two previous tokens first four characters of two next tokens last four characters of two previous tokens last four characters of two next tokens POS of two previous tokens POS of two next tokens whether the current token is in the “GEOG” dictionary
Finally, a dictionary of geography based on French is extracted from webpage[
After the features of token are generated, extracting an optimal subset from all the features is the most important step for building an effective classification model. At present, search algorithms can be divided into complete-based search, heuristic-based search and random-based search. The sequential forward selection (SFS) and sequential backward selection (SBS) based on heuristic are the most commonly-used algorithms for selecting features. Beginning with an empty feature subset X, SFS add a feature x into X, and ensure the optimal performance of evaluation function J(X). After n-times iteration, the classification model is constructed based on local optimum. Instead of SFS, SBS starts a full feature set, and eliminate a feature from the feature set for each iteration.
Compared with the above algorithms, we design and realize the bidirectional search (BDS) algorithm which combines the advantages of SFS and SBS, and improves the efficiency. The main idea of BDS is that SBS is used to search features, which is beginning with a full feature subset, while using SFS algorithm to search features beginning with an empty feature subset. Until a same feature subset is searched from both of SFS and SBS after n-iteration, BDS uses the same feature subset as the final results. After the selection step, the results for the different categories are shown in Fig. 1.
Furthermore, we list the Fmeasure of the intermediate result, which is generated either SFS or SBS, in the process of n-iteration. For each category of entity, the most optimal combination of feature can be selected by BDS as shown in Fig. 2. Although the method of feature selection may make out the local optimum, it can give better results than full feature subset for the feature selection of different entity categories. 2.4 Conditional Random Field The conditional random field algorithm is proposed by Lafferty in 2001. CRF is arbitrary undirected graphical model that bring together the best of generative models and Maximum Entropy Markov Models (MEMM). A potential function is defined as follow: yc ( yc ) exp(k fk (c, y | c, x))
k Where y ( yc ) is a potential function of the fully connected network of Y, which is c built on undirected graph. y | c represents random variables which correspond to the cth node in the fully connected network by boolean form. Given an observed sequence of tokens, x x1x2...xn , CRF can predicts a corresponding sequence of labels, y* y1 y2...yn . y* , which maximizes the conditional probability p( y | x) , is defined as follow: p( y | x)
1 Z (x) exp( k fk (c, yc , x)) cC k (1) (2)
The conditional random field algorithm is widely used in named entity recognition.
The existing open source toolkit CRF++[
Firstly, the training and testing data are generated based on the features. A CRF model can be learnt after training on the training data which is described in Sec. 2.1. Then the tokens in the testing data can be classified into one of the entity categories or non-entity category using CRF model. 3 3.1
The WI-ENRE system consists of two main modules, ten sub modules and one evaluation module. The purpose of this system is to automatically identify clinically relevant entities in medical text in French. One of the major components is the named entity recognition module, which can identify the clinical entity based on Conditional Random Field and generate the specific model for each category. In the pre-processing, the biomedical texts are divided into two parts: MEDLINE and EMEA. Then, using the CRF model to recognize the clinical entity, the results will be evaluated and determined whether the feature set should be optimized. Until the results meet the optimization condition, the CRF models will be stored in the model repositories. The second module integrated with UMLS can select the CUIs to map clinical entity, and generate the annotated biomedical texts automatically. Besides English, UMLS does not support the other languages, such as French, Chinese and so on. Therefore, the API of Google is used to translate the entities from French to English in the first step. Then the translated entities are put into UMLS and mapped to the CUIs which is selected with the first result.
In the part of named entity recognition, the first step is the preprocessing of the file,
which contains the part-of-speech tagging by Stanford Parser and the generation of
training files based on entity category. The next step includes the training of CRF model,
the decoding of CRF by testing files and the evaluation of entity results. Then the
module of feature optimization is performed until the optimum result is found. Finally, all
of the optimum model for each category will be stored into model repositories.
For task 1b, we determined the performance of WI-ENRE by comparing the system
outputs against reference standard annotations. The system performance and
performance for each category are evaluated rigorously. Precision, recall and Fmeasure[
Precision TP / (TP FP)
Recall TP / (TP FN) Fmeasure 2* Recall Precision / (Recall Precision) (3) (4) (5) 3.3
Using the evaluation metrics described above, the results of the WI-ENRE system are shown in Tab. 4 and Tab. 5.
The evaluation results of EMEA and MEDLINE are presented respectively. The experiments show that results of EMEA are better than MEDLINE. In the 10 main categories, GEOG based on lexicon get the high Fmeasure above 80 and 70 percent in different corpus. Compared to GEOG, the categories which are based on CRF, such as ANAT, PROC and LIVB, have a low Fmeasure about 70 percent.
In addition, the rest categories are worse than ANAT, PROC and LIVB, with below 50 percent. Through the analysis, it is observed that the entity categories of low accuracy do not basically select the orthographic features which are inside the feature range of 6th and 11th (as shown in Fig.1). Moreover, we also found that the entity categories which select the feature of POS get higher percentage of accuracy than others. 3.4
The errors in the WI-ENRE system are analyzed according to the error analysis
method[
According to the three groups of error, missing errors make up the highest proportion as 52.72%. Therefore, the recall of the WI-ENRE system is very low. ANAT CHEM DEVI DISO GEOG LIVB OBJC PHEN PHYS PROC Spurious total 2 1 3 2 2 124 10 2 3 1 6 58 1 3 11 4
DISO 1 2 1 17 22 340 43 7 0
The experiment shows that the categories of CHEM and DISO have high missing error with the count of 946 and 846, respectively. Twenty-two PROC entities are identified as DISO while 10 DISO entities are marked as PROC. It is difficult to distinguish between PROC and DISO for WI-ENRE. In addition, ANAT, LIVB, PHYS have a missing count of above 200. All of these led to the low recall rate of WI-ENRE system. Compare to missing errors, the spurious errors of DISO are also much higher than others. It follows that the system cannot recognize the category of DISO well, which not only has the higher missing errors but also is the most serious error of spurious. For the type error, a normal level which can be remained within acceptance criteria is shown in Tab. 7. 4
This paper described the clinical entity recognition by machine learning method for the task 1b of CLEFeHealth 2015. A suite of methods that included conditional random fields, feature selection with BDS algorithm and entity normalization using MetaMap performed the task well. Among these methods, the feature selection plays a crucial role to enhance the performance for each category. Using a suitable feature subset, we can obtain more accurate and reasonable classification than the full feature set. In order to testify this method, we design the system, WI-ENRE, to address the clinical entity based on CRF and achieve the normalization of clinical entity by UMLS.
The future study will be focused on the feature optimization and the improvement of recall rate. Moreover, the term vectors which are generated by word embedding can be taken as the characterizing attribute. The other useful features and more suitable methods will be researched to improve our system.
Acknowledgments. The MEDLINE title and EMEA documents used in this paper were provided by CLEFeHealth 2015 task 1b, and thanks to the organizing committee of CLEF and the annotators of the dataset.