Clinical name entity recognition is a task of CCKS2017. The purpose of this task is to recognize symptom, disease, exam, treatment and body words from medical records. In this paper, we propose two methods based on conditional random fields (CRFs) and LSTM-CRF. The experiment shows that our system is efective in the clinical name entity recognition of medical records, achieving a F1 measure of 0.8974 at the strict entity evaluation level which ranked sixth.
1.1
Task definition For a given set of electronic medical records, the goal of the task is to identify and extract the entity mention related to medical clinics and classify them into pre-defined categories, such as symptom, disease, exam, treatment and body.
The neural networks is widely used to bulid the state-of-the-art sequence labeling systems. We chose the network architecture combining of bidirectional LSTM and CRF. 2.1
Basic model The evaluation task gives the gold standard annotation data and the unlabeled data, given the entity location and category in the text. We use the char as a unit for sequence to modeling the text, deal with the entity recognition as a sequence labeling problem. Make this problem be a seq2seq model.
Neural Network Architecture In this section, we describe the components (layers) of our neural network architecture. We introduce the neural layers in our neural network one by-one from bottom to top.
B char1 char1
I char2 char2
E char3 char3
O char4 char4
Char Embedding We use the char level vector to represent the Chinese words,by using the word2vec in the unlabeled dataset, the embedding dimension is 100.
Bi-directional-LSTM The basic LSTM is designed to cope with the gradient vanishing problems of RNN.The formulas to update LSTM unit at time t are: it = (Wiht 1 + Uixt + bi) ft = (Wf hh 1 + Uf xt + bf ) ct = tanh(Wcht 1 + Ucxt + bc)
ct = ft⊙ct 1 + it⊙ct
ot = (Wohh 1 + Uoxt + bo)
ht = ot⊙ tanh(ct)
(
For many seq2seq tasks, it is beneficial to have access to both past (left) and future (right) contexts. The solution whose efectiveness has been proven by previous work (Dyer et al., 2015) is bi-directional LSTM (BLSTM). The basic idea is to present each sequence forwards and backwards to two separate hidden states to capture past and future information, respectively. Then the two hidden states are concatenated to form the final output. This step allows the hiden state to capture both past and future information.
CRF A limitation of the single LSTM architecture is that cannot make good use of the output information to get the label. For sequence labeling tasks, it is beneficial to consider the correlations between labels in neighborhoods and jointly decode the best chain of labels for a given input sentence.Therefore, we model label sequence jointly using a conditional random fields (CRFs) with existing LSTM (Laferty et al., 2001), instead of decoding each label independently. Normally, we use z = fz1; :::; zng to represent a generic input sequence where zi is the input vector of ith word. y = fy1; :::; yng represents a generic sequence of labels for z. (z) denotes the set of possible label sequences for z. The probabilistic model for sequence CRF defines a family of conditional probability p(yjz; W; b) over all possible label sequences y given z with the following form: p(yjz; W; b) = n ∏ i=1 i(yi 1; yi; z) n ∑ ∏ (yi′ 1; yi′ ; z) y2 (z) i=1 The i(y′ ; y; z) = exp(WyT′ ;yzi + by′ ;y) are potential function, and WyT′ ;y and by′ ;y are the weight vector and bias corresponding to label pair (y′ ; y). In our code, we try to use the negative log likelihood function and the labelwise function to get the loss of the CRF layer. We decode by using the marginal algorithm and viterbi algorithm.
For training example (x(t); y(t)) Log-likehood: Lablewise:
L(W; b) = Cjjwjj2 m ∑ log(Pw(y(t)jx(t))) t=1 m L R(W ) = ∑ ∑ Q(Pw(y(t)jx(t)) 2.3
Parameter Initialization The Parameter Initialization is(best performance): Randomly select 80% gold standard annotation to be the training data.And the rest to build the dev data set. 3
We also try to do the single CRF method. 3.1
Proprocessing Due to the limitations of the existing word segmentation tools in clinical medical text data, a large number of professional words and commonly used medical abbreviations are erroneously segmented, which in turn leads to a large number of boundary error entities in the entity recognition tasks. Therefore, we here cut the medical text based on the Chinese single word directly.
In order to get a better preprocessing result, we train the word segmentation model and part-of-speech(POS) tagging model separately based on the SVM algorithm with the CTB corpus. Also, we use the segmentation model and POS tagging model provided by LTP-Cloud to compare with our own models. 3.2
Feature Extraction Considered characteristics of the medical text entity, we extract a lot of features, including participle, part of speech, training set of physical dictionary and other characteristics. Finally, we only choose participle and part-of-speech as the valid training features.
For each word, we set length of the window as 5 and then extract features with single feature, 2-gram and 3-gram. 3.3
Model Merging In view that diferent training parameters would obtain diferent models and different models have their own advantages, we train several models using diferent parameters. Then we merge the models based on the cross-validation results of diferent categories to get the final model. We choose diferent model for diferent category which proform best on this category and merge them into a final result. 3.4
Training Parameters Here we only give parameters of the best main model, and the diferences on parameters of other support models are not obvious except for the Max_iterations. Here give the final best results of the online judge using LSTM-CRF and Single CRF:
Analysis of Result Compared these two results, we find that disease and treatment don’t have a high strict F-measure. Although treatment has a very high relaxed F-measure which means that we have found the right position of treatment without right boundary. The right answer is either too long or too short. As for disease, it seems that the disease entities we find are not much enough. For this issue, result may be better by including extra glossary dictionary. This can be a part of future work. 5
This work presents a system that is completely machinelearning-based; it uses neither a rule-based method nor a postprocessing module. In this named entity recognition task, we achieved a strict F-measure of 0.8974 which ranked sixth. Although we also use LSTM-CRF like other teams, we didn’t get a better score. Future work should attempt to adapt the parameter of the LSTM-CRF method in order to get a better F-measure. 6
The medical records used in this paper were provided by Jims-Cloud, and CCKS2017 Tasks were supported by the following Grants: Tsinghua Engineering Group, MSRA. Thank you to the organizing committee of CCKS and the annotators of the dataset.