EMR (Electronic Medical Record) refers to the systematized collections of patients' electronically-stored health information in a digital format. Clinical Named Entity Recognition aims to recognize and extract entity mentions from EMR. In this paper, we introduce a novel neural network architecture based on bidirectional LSTMs and conditional random fields, requiring no massive hand-crafted features or data pre-processing, compared to traditional statistical methods such as HMM and CRF.
Named entity recognition (NER) is one of the most important tasks for development of more sophisticated NLP systems. For news text, NER task has achieved relatively good performance. However, in other domains such as medical domain, there is still large gap. The main reasons for this gap are as follows: complicate and inconsistent terminologies, and ambiguities caused by abbreviations and acronyms.
CCKS2017 task2 is named entity recognition for electronic medical records, referred to as CNER. For a given group of electronic medical records (text file), the task is to identify and extract the related medical clinical entity names (entity mentions), and they are classified into pre-defined categories, including disease, symptom, examination, treatment and body part.
In numerous means of NER, CRF and LSTM are widely used. CRF (conditional random field) is a conditional probability model for marking ordered data, which combines the characteristics of the maximum entropy model and the HMM model. However, it can’t take the long term contextual information into account. LSTM model, which is powerful in sequence modeling, can capture long term context information.
According to this, in this paper, we present a novel model based on bidirectional
LSTMs and conditional random fields, which provides the best NER results ever
reported in standard evaluation settings, even compared with models that use external
resources, such as gazetteers.
We use a LSTM-CRF model (Guillaume Lample,Miguel Ballesteros,Sandeep
Subramanian,Kazuya Kawakami,Chris Dye.2001) to implement NER for Electronic
Medical Records. As shown in Figure 1, the model proposed in this paper contains four
components:
(
These components will be presented in detail in this section. In Chinese, as is known to all, word boundaries are not readily identified in texts. Word segmentation is a key first step to generate features for an NER system. Unfortunately, the result of word segmentation always has various mistakes. For instance, in the following example, the most frequent mistake is that the context and part of mention combine together. For example, in the labeled data set, word ‘上腹部’ is a mention, but character ‘上’ connects with its left context.
“以上/腹部/疼痛/为/主要/症状/。” To solve this problem, we investigate a new segmentation method named “ReSeg”. Firstly, we find out all of the clinical named entities appearing in the training dataset, and keep these mentions in sentence not being segmented. Then on this result, we segment these mentions again and require the final word segmentation result. And we'll analyze the effect of “ReSeg” in the experiments section.
we use the BIO format (Beginning, Inside, Outside) to represent sentences where every token is labeled as B-label if the token is the beginning of a mention, I-label if it is inside a mention but not the first token within the mention, or O which means otherwise. Furthermore, we follow the same format with the CoNLL2003 sharing task as the final input format for the training: each word has to be on a separate line, and there must be an empty line after each sentence. A line must contain at least two columns, the first one being the word itself, the last one being the named entity. Our input embedding contains two parts: character embeddings and word embeddings. We use pre-trained word embeddings to initialize our lookup table. Embeddings are pre-trained using word2vec and Glove respectively. Word embeddings are trained based on the training dataset and the unlabeled dataset in CCKS-2017 shared task-2. In detail, the embedding dimension is 100, window size is 8 and iteration times is 100. Recurrent neural networks (RNNs) are a family of neural networks that operate on sequential data. Though, in theory, RNNs are capable of capturing long-distance dependencies, in practice, they fail due to the gradient vanishing or exploding problems.
Long short-term memory networks (LSTMs) are variants of RNNs designed to solve these gradient vanishing or exploding problems. Basically, a LSTM unit contains three multiplicative gates which control the proportions of information to forget and to pass on in the next time step. Formally, the formulas to update an LSTM unit at time t are: = ( + ℎ −1 + −1 + ) = ( − ) ⊙ −1 + ⊙
( + ℎ −1 + ) = ( + ℎ −1 + + ) = ⊙
( )
Where σ is the element-wise sigmoid function, and ⊙ is the element-wise product. represents current input; represents a input gate with corresponding weight matrix , ℎ , , ; represents a output gate with corresponding weight matrix
, ℎ , , ; −1 represents the state generated in previous step; represents current state.
The representation of a word using this model is obtained by concatenating its left and right context representations. These representations effectively include a representation of a word in context, which is useful for numerous tagging applications.
Instead of modeling tagging decisions independently, we model them jointly using a conditional random field (Lafferty et al., 2001). 2.4
Consistency check Consistency check is a method of checking whether the predicted label class is consistent with the label class in training dataset. We have statistically evaluated 2399 different mention-label pairs in the training data. If the predicted label class is not the same as the original one in training dataset, we will manually modify the label to original label. For instance, the label of word “腔隙性脑梗死” in predicted sentence is “症状和体征”; but in training dataset this word is labeled as “疾病和诊断”. In this case, we change the word’s label to original label.
More formal, we denote the mention and label in predicted sentence by = We test our model on the dataset provided by CCKS 2017 CNER task. These data are electronic medical records. There are four different file folders and each one contains 300 labeled files and 2605 unlabeled files.
We use these data to randomly get train dataset (80%) and dev dataset (10%). And we can also get test dataset (10%). 3.2
Baseline In this section, we use the same dataset (CCKS-2017 shared task-2) and 4 different word segmentation tools to compare the performance of three common models without any pre-processing or feature engineering. One of models is traditional statistics ( 3 3.1
, ∀ (
), and .
Dataset
= ≫ = (
. .
, = ⋀
. .
) in training dataset respectively.
≠ . ) methods based CRF (conditional random field). One of them is the single neural network architecture –LSTM (Long Short Term Memory Unit). And the last one is the combination of two previous models of LSTM and CRF. In addition, we randomly initialize the word embeddings of LSTM.
CRF
LSTM LSTM-CRF In Table 1, LSTM-CRF model with hanlp tool obtains better performance than others. Therefore, we choose the LSTM-CRF with hanlp as the baseline model. 3.3
Results and Discussion Error propagation is a difficult challege in NLP tasks, particularly in Chinese text processing. Word segmentation is a key first step to generate features for Chinese NER system. To reduce the mistakes created by word segmentation, we propose a method named “ReSeg”.
In order to analyze the influence of “ReSeg”, we use the “ReSeg” method for each model. Parameter settings are consistent with 3.2.
In Table 2, we observe that the performence of each model is improved after using “ReSeg”. Especially on LSTM-CRF model it yields nearly 2% absolute improvement. The result shows that “ReSeg” is indeed useful and has no model restrictions, in other words, every model could use the “ReSeg” method.
In Table 3, We observed that pretraining our word embeddings gave us the biggest improvement in overall performance of +3.28 in F1. The dropout gave us an increase of +1.43 and finally learning character-level word embeddings resulted in an increase of about +0.51. The Consistency Check layer gave us an increase of +2.28. 4
In this paper, we propose a novel LSTM-CRF method to solve named entity recognition for electronic medical records. A key aspect of our model is that we combine traditional statistical model with neural network. The proposed model is very robust and it achieves better performance without using any external resources. 5 6
This work is supported by the National Natural Science Foundation of China (NSFC No. 61572120, 61300097and 61432013). We thank all anonymous reviewers for their constructive comments.