As a fundamental task in information extraction, named entity recognition (NER) has received constant research attention over the recent years. The 2018 China conference on knowledge graph and semantic computing (CCKS) challenge sets up a task for clinical named entity recognition (CNER). In this task, we presented a neural network ensemble approach, which combines five individual neural network models (i.e., CNN-CRF, BiLSTM-CRF, BiLSTM-CNN-CRF, BiLSTM+CNN-CRF and Lattice LSTM). In this approach, the various features (i.e., stroke, word segmentation and dictionary features) are adopted. On the official test set, our best submission achieves the F-scores of 88.63% and 95.19% under the “strict” and “relaxed” criteria, respectively.
In recent years, the medical information processing has become a popular research
focus as the generation of larger amount of electronic medical records and the
potential requirements for medical information services and medical decision supports.
As a fundamental task for medical information extraction, clinical named entity
recognition (CNER) has received much attention and has been organized as a
sharedtask in many challenges [
In the previous work, the state-of-the-art CRF-based NER methods depend on
effective feature engineering, i.e., the design of effective features using various NLP
tools and knowledge resources, which is still a labor-intensive and skill-dependent
task. Recently, deep learning has become prevalent in the machine learning research
community. For the NER task, several similar neural network architectures [
In this paper, we describe our CNER method for the CCKS-2018 CNER task. In our method, five individual neural network models (i.e., CNN-CRF, BiLSTM-CRF, BiLSTM-CNN-CRF, BiLSTM+CNN-CRF and Lattice LSTM) are used for CNER. Afterwards an ensemble model is built by combining these models’ results with majority voting. Moreover, we explored the effect of additional features to further improve the performance. 2
In this section, our approach for CNER is described. Fig. 1 shows the processing flow of our method. Firstly, some preprocessing steps including sentence splitting, word segmentation and stroke generation are performed. Secondly, a character embedding is learned with large amounts of unlabeled data by the cw2vec tool1. Moreover, the additional features (i.e., word embedding, dictionary feature and stroke feature) are introduced into the model. Then with the embeddings as input, five neural network models are trained by the annotated training set. Finally, the results from these models are combined by majority voting and entity type ensemble. The detailed description of our method is presented in the following sections.
The character embedding is used as the basic features of our method since the
character-based methods outperform word-based methods for Chinese NER [
To achieve a high-quality character and word embeddings, we collected a total of 3,005 clinical texts from the CCKS-2017 challenge. Then these texts and all CCKS2018 CNER training set (a total of 600 texts) were spilt into the words with the HanLP tool2. Afterwards, these data were used to train 100-dimensional character embedding and 50-dimensional word embedding by the cw2vec tool as pre-trained character and word embeddings.
Due to the complexity of the natural language and the specialty of the clinical 1 https://github.com/bamtercelboo/cw2vec 2 http://hanlp.linrunsoft.com/ medical domain, some linguistic and domain resource features can be employed to improve the performance of our model. We also explored the effect of dictionary and stroke features for our neural network models. For the dictionary feature, we used Sogou dictionary 3 to generate our drug dictionary feature. First, longest possible matches between the character sequences and dictionary entries were captured. Then, for each character in the match, the feature was encoded in BIOES tagging scheme. At last, a lookup table was used to output 50-dimensional dictionary embedding. For the stroke feature, we first obtained the stroke sequence of every character with HanDian4. Then a stroke lookup table containing a stroke embedding for every stroke was initialized randomly. Then the stroke embedding sequence of the character was fed into a convolutional layer. At last, an attention pooling layer is used to extract global features from the convolution layer as the stroke feature of the character. 2.2
In this section we describe in details the five individual neural network models (i.e.,
CNN-CRF, BiLSTM-CRF, BiLSTM-CNN-CRF, BiLSTM+CNN-CRF and Lattice
LSTM) used in our ensemble. These models have the similar neural network
architecture, i.e., the neural network with a conditional random field layer (NN-CRF).
The architecture of the NN-CRF model is illustrated in Fig. 2.
CNN-CRF model. In the convolutional neural network (CNN) with a CRF layer
model, a convolution operation is applied to produce local features. Firstly, a
sentence is represented as a sequence of embeddings. Next, the embeddings are given
as input to a CNN layer. Then a tanh function on top of the CNN layer is used to
learn higher features. Finally, the CRF layer is added after the tanh layer to predict
the best label sequence path in all possible tag paths.
BiLSTM-CRF model. We also employed a bidirectional long short-term memory
with a CRF layer (BiLSTM-CRF) model for the CNER. Firstly, a sentence is
3 https://pinyin.sogou.com/dict/detail/index/270
4 http://www.zdic.net
represented as a sequence of embeddings. Next, the embeddings are given as input to
a BiLSTM layer. In the BiLSTM layer, a forward LSTM computes a representation
of the sequence from left to right, and another backward LSTM computes a
representation of the same sequence in reverse. These two distinct networks use
different parameters, and then the representation of a word is obtained by
concatenating its left and right context representations. Then a tanh function on top of
the BiLSTM layer is used to learn higher features. Finally, the CRF layer is added
after the tanh layer to predict the best label sequence path in all possible tag paths.
BiLSTM-CNN-CRF model. Different with the above models, the NN layer of
this model is a BiLSTM-CNN layer. In the BiLSTM-CNN layer, a BiLSTM
computes a representation of the sequence, which is then fed into a CNN layer to
learn higher features.
BiLSTM+CNN-CRF model. The model is similar with the BiLSTM-CNN-CRF
model, but the BiLSTM-CNN layer is replaced with the BiLSTM+CNN layer. In the
BiLSTM+CNN layer, the representation of the previous layer is fed into a BiLSTM
layer and a CNN layer, and then their outputs are concatenated and fed into a tanh
layer.
Lattice LSTM model. Recently, Zhang and Yang proposed a lattice-structured
LSTM model for Chinese NER [
As introduced before, five machine learning-based methods were deployed for the CNER independently. To take the advantages of different methods, we used a majority voting approach to combine all predicted entities. In addition, different combinations of the models were also investigated. Finally the models with the best performances for the five entity types (i.e., anatomy, symptom, independent symptom, drug and operation) on the development set were combined to output the final result. 3 3.1
In the CCKS-2018 CNER challenge, organizers provided a corpus including the training and test sets. The training set consists of 600 medical records annotated with five categories of entity, including anatomy, symptom, independent symptom, drug and operation. And the test set consists of 400 medical records. In our experiments, we randomly selected the 20% of the training set as the development set to tune the hyper-parameters. The statistics of entity on different categories are list in Table 1.
Dataset Training set Development set The evaluation metrics of this task include two criteria: 1) strict metrics which define a correct match as that the ground truth and extraction result share same mention, same boundaries and same entity type; 2) relaxed metrics which only consider the ground truth and the result have same entity type and overlap boundaries. All our evaluations were performed on the official test set using the evaluation tool of CCKS2018 CNER challenge, which outputs micro-average precisions (Prec.), recalls (Rec.) and F-scores (F) via the strict metrics. 3.3
In this task, we directly divided the sentences into Chinese characters, which can avoid the boundary error of entity caused by the word segmentation tools. The “BIOES” (i.e., B-begin, I-inside, E-end, S-single, O-outside) tags are used to represent the entity. Table 2 shows the performance of various models on our development set. Our basic model is BiLSTM-CRF which achieves an F-score of 90.88%. When stroke and dictionary features are added into the basic model, the model achieves performance improvement. The BiLSTM-CRF+Stoke+dic achieves an F-Score of 92.64%. It demonstrates that the features can help boost the performance of the model.
In addition, among the various models, BiLSTM-CRF+ALL and BiLSTM-CNNCRF+ALL perform better than others for anatomy entities. BiLSTM-CRF+ALL and BiLSTM+CNN-CRF+ALL perform better for drug entities. To take the advantages of different methods, we used a majority voting approach to combine all predicted entities. The results show that the model ensemble achieves the highest F-score of 93.16%.
Table 3 lists the results on the official test set. Our best submission achieves the Fscores of 88.63% and 95.19% under the “strict” and “relaxed” criteria, respectively. We analyzed the results and found that the anatomy and operation have poor performance. Therefore, how to recognize the anatomy and operation entities more accurately will be the main focus in our future work.
F In this paper, we present a neural network ensemble approach to automatically recognize clinical entities from Chinese clinical texts. In this approach, five different neural network models are explored, and the ensemble can achieve better performance. In addition, the effect of additional features for these models in the CNER task is also explored. The experimental results show that the additional features are effective to improve the performance of our system. At last, our best submission achieves the Fscores of 88.63% and 95.19% under the “strict” and “relaxed” criteria on the official test set, respectively. We will focus on the more effective extraction of anatomy and operation entities in the future work. 5