With rapid development of electronic medical records, more and more attention has been attracted to reuse these data for research and commercial. As the entity recognition is one of the most primary task for medical information extraction, the 2017 China conference on knowledge graph and semantic computing (CCKS) challenge sets up a track for clinical named entity recognition (CNER). The organizers provide 400 annotated Chinese medical records for this track, 300 out of them are used as a training set and 100 as a test set. Other 2,605 raw medical records are released as an unlabeled set. In this study, we develop a hybrid system based on rule, CRF (conditional random fields) and RNN (recurrent neural network) methods for the CNER task. Experiments on the official test set show that our system achieves the F1-scores of 91.08% and 94.26% under the “strict” and “relaxed” criteria respectively, ranking first in the 2017 CCKS CNER challenge. By applying a self-training method with unlabeled data, the F1-scores of all machine learning-based methods are improved by about 1.0% under “strict” criterion. The future work of us will focus on the more effective extraction of body, disease and treatment entities.
In recent years, the medical information processing has become a popular researching
focus as the generation of larger amount of electronic medical records and the
potential requirements for medical information services and medical decision supports.
Clinical entity recognition, one of the most primary clinical text processing task, has
been organized as a shared-task in many challenges, such as the i2b2 (the center for
informatics for integrating biology & the beside) 2009[
The early clinical entity recognition systems are mainly based on
dictionarymatching and rules, such as MedLEE[
In this study, we participate in the 2017 CCKS CNER challenge and develop a hybrid system for the Chinese clinical entity recognition, which is based on four individual methods (rule, CRF, RNN and RNN with features) and a vote-based approach. Besides, we also apply a self-training method with a large unlabeled dataset to improve the performance of our system. 2
Figure 1 shows the overview architecture of our system for the entity recognition from Chinese clinical text, which contains four individual methods: rule-based, CRFbased, RNN-based and RNN with features methods. Firstly, we deploy these methods on Chinese clinical text respectively, the results of rule-based method are used as the features in other three methods. Then a vote-based method is used to combine all predicted entities by them. The detailed description of our system is presented below.
Since the prior-knowledge plays an important role in the entity recognition, especially for the clinical text, we construct several dictionaries for each type of entity referring to the training set and some open websites (e.g. “Baidu baike”, “Xunyiwenyao”, etc.), such as: body location, disease, symptom, examine, surgery, medicine, etc.
Assisting by these dictionaries, we build lots of rules to recognize the common patterns of entities, for example, in the phrase of “右侧小脑” (“right epencephalon”), “小脑” (“epencephalon”) can be identified as “Body” by dictionary-matching, the “右 侧” (“right”) will be extended by our rules. Besides, in “…有心脏病病史…” (“...has history of heart attack…”), we can extract the “心脏病” (“heart attack”) as “Disease” according to the pattern “…有…病史…” (“…has history of …”). 2.2
As the CRF algorithm has been widely used for sequence labeling tasks, we also develop a CRF-based method for the clinical entity recognition using CRF++ as the implementation of CRF. The features used in this method include: n-gram, radical feature, spelling feature, word segmentation, part-of-speech, section head, dictionary feature, relation feature, distributed representation of word, rule feature, etc. 2.3
In this paper, we also employed a bidirectional LSTM (BI-LSTM), long short-term memory - a variant of RNN, method for the entity recognition from Chinese clinical text, which consists of three main layers: 1) input layer, generates the representation of each word in a sentence; 2) LSTM layer, takes the word representation sequence as input and generates a new one that captures the context information of the words. It contains both forward and backward LSTM networks; 3) output layer, learns the dependencies between successive labels by a transition matrix, and predicts a best label sequence according to the output of LSTM layer.
To utilized the hand-crafted features constructed in above rule and CRF based models, we extend above neural network (BI-LSTM) by adding a hidden layer (fully connected layer, FC) after the LSTM layer to concatenate the feature representations, which is represented as BI-LSTM-FEA, as shown in Figure 2.
As introduced before, three machine learning-based methods are deployed for the clinical entity recognition independently. To take the advantages of different methods, we use a vote-based approach to combine all predicted entities by them: a candidate entity is selected only when it has been exactly predicted by at least two methods.
Except the annotated data, the organizers of 2017 CCKS challenge also provide a set of unlabeled records. To explore the contribution of unlabeled data for the clinical entity recognition, we use a self-training approach, as follow: 1) train all individual methods on the official training set; 2) tag unlabeled records by above methods respectively, and combine all results by voting; 3) merge the tagged unlabeled data with the official training data as a new larger training set; 4) finally, retrain all above individual methods on this new training set, and tag on the official test data. 3 3.1
In the 2017 CCKS CNER challenge, organizers provided 400 medical records
annotated with five categories of entity (as in Table 1), 300 out of them are used as a
training set and 100 as a test set. Besides, other 2,605 unlabeled records are released as an
unlabeled set. The statistics of entity on different categories are listed in Table 1.
All our evaluations are performed on the official test set using the evaluation tool of
2017 CCKS CNER challenge, which outputs micro-average precisions (Prec.), recalls
(Rec.) and F1-scores (F1) under two criteria: “strict” - checks whether the boundary
and category of an entity is exactly matched with a gold one; while “relaxed” - only
considers the boundary of an entity is overlapped with a gold one of same category,
“strict” is the primary one.
In this study, we directly divide the sentences into Chinese characters, which can
avoid the boundary error of entity caused by the word segmentation tools. The
“BIOES” (B-begin, I-inside, E-end, S-single, O-outside) tags are used to represent the
entity. For neural network models, we use the stochastic gradient descent (SGD)
algorithm to estimate parameters, and the pre-trained Chinese character embedding was
learned from training and unlabeled datasets by word2vec tool. The feature
representations are randomly initialized from a uniform distribution ranging in [
Table 2 shows the performance of various methods on test set. We can see that all the machine learning-based methods outperformed the rule-based method, BI-LSTM model achieves much better F1-scores than CRF-based method, the voted results (90.17% under “strict” criterion) outperformed all the individual methods. After applying the self-training approach, the F1-scores of all individual methods are improved by about 1.0% under “strict” criterion, and the F1-score of voted result is improved by 0.06%. However, the BI-LSTM-FEA model performs much poorly than other methods, which performs best on our validation set (divided from training set). We think that the bad results of rule-based method on test set may cause the poor performance of BI-LSTM-FEA model. In this study, we proposed a hybrid system based on rule, CRF and RNN methods for the entity recognition from Chinese clinical text. Experiments on 2017 CCKS corpus show that our system achieves the F1-scores of 91.08% and 94.26% under “strict” and “relaxed” criteria respectively, ranking first in this challenge. Among all individual methods, BI-LSTM outperforms rule-based and CRF methods. By applying a selftraining approach with unlabeled data, the F1-scores of all machine learning-based methods are improved by about 1.0% under “strict” criterion. The future works of us will focus on the more effective extraction of body, disease and treatment entities.
This paper is supported in part by grants: National 863 Program of China (2015AA015405), NSFCs (National Natural Science Foundations of China) (61573118, 61402128, 61473101, and 61472428), Special Foundation for Technology Research Program of Guangdong Province (2015B010131010), Strategic Emerging Industry Development Special Funds of Shenzhen (JCYJ20140627163809422, 20151013161937, JSGG20151015161015297 and JCYJ20160531192358466), Innovation Fund of Harbin Institute of Technology (HIT.NSRIF.2017052), Program from the Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education (93K172016K12) and CCF-Tencent Open Research Fund (RAGR20160102).