Named Entity Recognition is one of the subtasks of Natural Language Processing, which aims to locate and classify named entities in text into pre-defined categories. The CANTEMIST 2020 is a task for tumor named entity recognition and we propose a new model for this task. We use Recurrent Neural Networks and Convolutional Neural Networks to extract relevant text features. Then dynamic attention mechanism is used to merge features extracted from these two structures. In the final evaluation, we achieve an F1-score of 0.746.
Cancer is one of the diseases with the highest mortality rate, and the number of people sufering from cancer are increasing year by year. With the rapid development of natural language processing (NLP), it has become an essential means of early cancer diagnosis, which can efectively prevent the deterioration of the tumor. The BNER task is a prerequisite task for other NLP tasks on cancer, and its performance significantly afects the final diagnosis. Tumor named entity recognition is a subtask of BNER in the tumor field, and its experimental method is similar to the BNER task.
The current main methods for BNER are machine learning methods and deep learning
methods. For example, Makino et al. [
With the rapid development of Neural Networks (NN), people have gradually abandoned
traditional methods to deal with BNER tasks and replaced them with NN methods. Sahu et al.
[
Our model includes four parts: the embedding layer, the parallel layer, the attention structure, and the classification layer. In the embedding layer, we vectorize the text sequences by using word embedding and character embedding. The RNN and CNN modules are used to extract context information respectively in the parallel layer, and then we use the dynamic attention structure to merge these contextual information. Finally, the dense layer acts as a classifier to output the final prediction. The overall model structure is shown in Figure 1.
In order to alleviate the OOV problem, our model combines word embedding and character
embedding to obtain OOV information in the embedding layer. The character embedding is
initialized by random feature vector, while word embedding is initialized by Spanish Medical
Word Embeddings [
For the word embedding, we denote the input sentence as = [ 1, ..., , ..., ], where is the i-th token, as well as is the number of token in the sentence. The word embedding is expressed as follows: =
( ), to supplement word embedding to obtain OOV information. The character embedding is () is the word embedding function. Then we use character embedding = [ 1, ..., , ..., ],
= ℎ ( ), .
where ℎ is the number of characters in the token , and represents the j-th character in the token . Because the dimensions of word embedding and character embedding are inconsistent, they can not be merged directly. Here we use CNN to fine-tune the character embedding, and the () is a character embedding function to randomly initialize the token , specific step is as follows: =
([ − /2, .., , ..., +( /2)]), where represents the size of the convolution kernel, and k is set to 3 in this paper. Finally, we concatenate word embedding
and character embedding to obtain the final token features The parallel layer is aimed to extract the context-dependence information of the text sequence. It consists of an RNN module to obtain the long-distance dependence information and a CNN module to obtain the local dependence information.
RNN module: Whether in industry or academia, long short-term memory (LSTM) [
( ⋅ [ ∗ −1, ]), = (1 − ) ∗ −1 ∗ ̃ , where , , and are parameter matrix for calculating the update gate , the reset gate , and the new memory ̃ , and then we can get the output = [ 1, ..., , ...] of the GRU. In order to extract long-distance dependence information more comprehensively, the model stacks the GRU and the SRU. At the same time, considering the overfitting caused by ordinary stacking and the disappearance of SRU gradient, the output of GRU is added as a penalty to the output of SRU. After the addition operation, the output of the final RNN module is as follows: = ( + −1 + ), = −1 + (1 − )( ), = ( + −1 + ), = + (1 − ) ,
= + , where , , are parameter matrix, and , , and are parameter vector in SRU. = [ 1, ..., , ...] is the final output of the entire RNN module.
CNN module: Because of the characteristics of convolution, CNN can not capture the longdistance dependence information in the text sequence. However, due to the window sliding mechanism in convolution operation, we can obtain obvious local features by controlling the size of convolution kernel. In the CNN module, it extracts local information between diferent distances by setting three convolutional layers with diferent kernel sizes. And through a lot of tuning experiments, we set the size of the three convolution kernels to 3, 5 and 7 respectively. Then we perform data compression through average pooling operation while reducing data redundancy. Finally, the results of three convolution layers are added to get the output of CNN module
Similar to the self-attention mechanism [
+ ), = ⋅ + (1 − ) ⋅ . where is weight and is bias term. Through the weight control of the activation function, the attention matrix of the current sequence can be obtained after addition operation. Taking into account the new noise brought by the addition operation, we execute the activation function on , which can enhance the weight of the efective value: where is weight and is bias term. Then we can calculate the attention matrix and use it to weight the output of the RNN and CNN modules. The specific method is as follows:
Through the attention structure, our model can assign diferent weights to the output of the RNN and CNN modules, and filter the useless information of these two modules at the same time, thereby efectively reducing the impact of redundant context-dependence information on the final result. 3.3.1. Classification layer After encoding at the parallel layer and the attention structure, the model obtains a sequence of semantic features that contain context-dependence. Considering the over-fitting problem during training, only a single dense layer is used as the final classifier. Finally, through the activation function which computes the probability of a label for each tag, the final prediction is obtained.
The CANTEMIST corpus consists of 3000 clinical cases, and the professional clinical coding experts annotate these clinical cases in Spanish with eCIE-O-3.1 codes using the BRAT annotation tool. These cases are distributed in plain text in UTF8 encoding, and each clinical case is stored as a file. The corpus is randomly divided into training set, development set and test set. There are two development set in this task, and we use them to adjust the hyperparameters of this model. The test set have 300 clinical records with standard annotations, which can be used to test the performance of the model. This task also introduces a background set without standard annotations to prevent the participating teams from making manual corrections. Finally we count the number of records and sentences to help understand the distribution of the data sets. The results are shown in Table 1.
In the preprocessing of this task, we perform basic sentence and word segmentation operations on the original text. Sentences in clinical texts are split by using specific punctuation (such as line breaks or periods) and words are split by using spaces. Then taking into account the character embedding in the model, the original text was not processed with stop words in the preprocessing, and we also remove punctuation marks, special symbols, and single-word sentences.
Finally, we use Keras and Tensorflow as the model backend to implement the model, and train and predict on NVIDIA GeForce GTX 2080Ti. The detailed hyperparameter settings of the experiment are shown in Table 2.
To verify the stability of the model and the credibility of the result, we compare our model with the BERT, which is the most popular model for NER. The result shown in Table 3. It shows model
P model GRU SRU
LSTM GRU+LSTM GRU+SRU
P that our model performance on this data sets is better than the BERT, and the F1-score exceeds 2%. Compared with the BERT, our model parameters are less and the training cost is lower. The comparative experiment proves that our model performs well on the tumor named entity recognition in Spanish.
The ablation study illustrates the efectiveness of the RNN and CNN modules in this model.
For the RNN module, we mainly compare the performance of LSTM, GRU and SRU without stacking, and the impact of diferent stacking methods on model performance. Firstly, we use LSTM, GRU, and SRU as encoders to encode the embedding layer features without stacking. From experiments, we find that GRU and LSTM have similar performance on this task, and both F1-scores are 0.736. Consider the cost of training, GRU can efectively reduce single training time and the required parameter is also less than LSTM. Therefore, we use the GRU as the first encoder of the RNN module. For the second encoder of the model, the module of SRU and LSTM have similar performance when stacking GRU, with F1 scores of 0.746 and 0.745 respectively. Because SRU reduces a lot of parameters and training costs, so combination of SRU and GRU will be a more suitable choice for this task. The RNN module comparative experiment results are shown in Table 4.
For the CNN module, we want to prove whether the existence of the CNN module is necessary. Firstly, we use the RNN module directly stack with the embedding layer, and the F1-score of the model can only achieve 0.727. Then we use the CNN module to replace the embedding layer and find that the F1-score of the model achieve 0.746. This shows that the CNN module is helpful to improve model performance. The experiment results are shown in Table 5.
model RNN+embedding
RNN+CNN
P
In this paper, we verified the efectiveness of the parallel-attention model and the outstanding performance of RNN on tumor named entity recognition through CANTEMIST-NER task. Compared with BERT, which has higher training costs, over-reliance on corpus, and risk of overfitting, our model is more lightweight. This also shows that RNN is a more suitable choice for NER tasks.
In our model, we use the local dependency extraction ability of CNN to make up for the feature extraction ability of the RNN. At the same time, the long-distance dependence extraction ability of RNN is further strengthened by stacking the GRU and SRU. Finally, by adding a dynamic attention mechanism, our model can filter the redundant features extracted by the CNN and RNN. For the OOV problem, we combine character embedding and word embedding to alleviate this problem. And the comparative experiments and ablation study are also suficient to show the outstanding performance of the model and the rationality of each module in this model.
For future work, our team hopes to compare the model with more advanced models and verify the efectiveness of the model on more data sets to improve generalization capabilities.