Named entity recognition is to recognize the mention of a certain thing or concept in a text in natural language processing, and it is the basis of many natural language processing tasks such as relation extraction, event extraction, knowledge graph, machine translation, and question answering systems. This paper describes the solution of CANTEMIST's named entity recognition subtask. The core idea of the method is to process it as a sequence labeling task and uses a neural sequence model to solve it. We use a pre-trained language model for semantic feature embedding, a recurrent neural network for semantic inference, and a label-based attention mechanism to predict the output. In the final test, our F1 score is 0.719.
At present, cancer has become one of the major diseases endangering human health. About 9 million people die of cancer every year in the world. It is very urgent to find an efective treatment, but medical resources are dificult to meet the needs in the face of a large number of patients. In a large number of individual clinical records of cancer patients, extracting information and knowledge from them can provide help for curing cancer patients. However, in the face of these free-text clinical records produced by professional doctors, eficient extraction of information and knowledge is a very challenging task. Natural language processing(NLP) technology is a very efective method to process individual clinical records. It can extract information from unstructured or semi-structured data to form structured data for further research.
The Cantemist [
1. CANTEMIST-NER: finding mentions of tumor morphology in oncology cases. 2. CANTEMIST-NORM: recognition and mapping to concept identifiers from ICD-O-3. 3. CANTEMIST-CODING: oncology clinical coding (multi-label classification) assigning
ICD-O-3 codes to clinical case documents.
Our team participates in the CANTEMIST-NER subtask. Named entity recognition(NER) is the basis of NLP tasks such as relation extraction, event extraction, knowledge graph, machine translation, and question answering systems. This subtask is aimed to identify related entity in tumor morphology. A tumor refers to the growth or formation of new abnormal tissues. The tissue and cell types that make up the morphology usually determine the expected growth rate, the severity of the disease, and the recommended type of treatment. We deal with NER using a neural sequence model that combines a pre-training language model with a multi-head attention mechanism, and our model achieves good result in the final evaluation.
NER of clinical text is a very popular research direction, and the categories of recognized named
entities are also very large, such as disease, patient, symptom, drug, body state, etc. At present,
there are four research methods: rule-based method, dictionary-based method, traditional
machine learning method, and deep learning method. The rule-based method [
Thanks to the growth of computing resources, deep learning becomes a competitive method
to solve NLP problems. The deep learning method also regards the NER task as a sequence
labeling task, which can avoid complex feature engineering compared with traditional machine
learning method. For biomedical NER tasks, various deep learning models have achieved a
pleasing performance. There are two main aspects worth paying attention to: The first aspect
is how to improve the reasoning ability of the model. For example, the Bi-LSTM-CRF [
In this section, we describe the methods used in the task. Our model is a neural sequence labeling
model, which consists of a semantic representation module, semantic reasoning module, and
output module. As shown in Figure 1: the semantic embedding module of the model is a
pre-train language model similar to BERT [
The goal of the CANTEMIST-NER task is to automatically find tumor morphology mentions in the clinical records of tumor patients. The specific form is to find the corresponding character ofsets of tumor morphology mentions in the given UTF-8 plain text medical document. The entire corpus contains a total of 5982 medical documents, of which 1301 are annotated and 4681 are unannotated. Experts annotate some medical documents by referring to the labeling guidelines that contain rules for annotating morphological tumors in Spanish tumor clinical cases and map these annotations to CIEO-3. All entities associated with tumor morphology are labeled MORBOLOGIA_NEOPLASI type, which include the tumor name, anatomical location, histological morphology, and other information. The distribution of the corpus is shown in Table 1.
Semantic representation plays a very important role in the NLP system. Good representational
engineering can efectively improve the performance of the system. We choose BETO [
In the semantic reasoning module, we use a Recurrent Neural Networks (RNNs) to extract more advanced semantic features. The advantage of RNNs is that it can capture longer distance dependence when processing sequence information, compared with Convolutional Neural Networks (CNNs) and Deep Neural Networks (DNNs). Gated Recurrent Neural Networks (GRU) and Long Short-Term Memory (LSTM) are two variants of RNN, and both GRU and LSTM are proposed to solve the problem of gradient disappearance or explosion in backpropagation. However, compared with LSTM, GRU can reduce the amount of computation while achieving the same efect.
The multi-layer GRU stack is used to further extract semantic information from the se{ ; 2 ; ...; mantic representation module. For a sentence, the embedded semantic representation = } of the sentence is obtained after the semantic representation module. Then the embedded semantic representation is fed into the stacked bidirectional GRU layer to obtain a Finally, the two hidden states are concatenated as the final sentence representation. sequence of forward hidden states {ℎ⃗1; ℎ⃗2; ...; ℎ⃗ } and backward hidden states {ℎ1; ℎ2; ...; ℎ }. ← ← ← ℎ = [ℎ⃗ , ℎ ]
← = {ℎ1; ℎ2; ...; ℎ }
During the label decoding phase, we use a hierarchically-refined label attention network [
= { 1, 2, ..., | |} are embedded in the label represen = ( ) The representation space of the label is to randomly initialize the embedded vector of each label, matrix form of the standard attention mechanism is as follows: where is to map each label to the representation space by looking up the table.
Key-value pairs are used to calculate the distribution of attention in the multi-head attention. The sentences output by the semantic reasoning module represent ℎ as the query vector, and the key vector and value vector are label embedded vector , namely = = . The = =
(, , max( √ ) ) = where ∈ ℝ × ,
∈ ℝ ×| |. And and ℎ are the sentence length and the hidden state size of GRU respectively, and the embedding dimension of the label representation feature must equal to ℎ. The | | is the total number of labels. Compared with the standard attention mechanism, the interaction of the multi-head attention mechanism can efectively infer multiple potential label distributions in parallel. The forward propagation of multi-head attention is as follows: = ℎ = (ℎ 1, ℎ ( , 2, ...ℎ , ) + ) that can be learned in training. is the number of parallel heads. where ∈ ℝ × ,
∈ ℝ × , and ∈ ℝ × are a parametric matrix of linear transformation
If it is not the last layer of the label attention network, the hidden state of the GRU and the output of attention are connected as the final output: = [ , ] The last layer of label attention network directly outputs the weight matrix in the attention mechanism as label prediction: where ̂
represents the probability that the -ℎ word is the -ℎ label, and ̂ represents the predictive label of the -ℎ word in the sentence.
This section introduces the experimental of the model, mainly including data preprocessing, experimental setting, and error analysis.
In the data preprocessing stage, we divide the text according to the sentence level and word level. Then, according to the annotation file, all the sentences are labeled in the format of BIO. After getting the labeling sequence of each sentence, we clean the sentence. Specific operations include deleting all punctuation marks and special characters; replacing numbers with "Cifra"; deleting single-word sentences (mainly subtitles). As shown in Figure 2, the sentence distribution after preprocessing is counted to provide a reference for the model’s sentence length hyperparameter setting. As can be seen from Figure 2, most sentences are less than 80 in length. Considering the Bert model, a masked language model is used for training. In the masked language model, 15% of the words are divided into several unknown words, and the unknown words are predicted by other words in the sentence. In the semantic embedding module of our model, the hyperparameter of sentence length is set to 100 to reduce the noise caused by excessive zero paddings.
This task is the same evaluation as most NER tasks, using the F1 score as an evaluation metric. Because this task only defines one type of entity, there is no data imbalance problem. The micro-average F1 score can be used to efectively evaluate the system performance. The calculation formula is as follows:
− ( ) = ( ) = ( 1) = 2 + + × +
According to Section 3.4, if the label attention network has only one layer, it is equivalent to using the Softmax layer. So we compare the performance of the label attention network with diferent layers. When the number of layers of the label attention network is set to 1, 2, 3, 4, and the results on the development set are shown in Table 2. We find that the performance of the development set can be improved, as the number of layers of the tag attention network increases. However, when the number of layers is increased to 4, the result is not improved. Besides, the embedded dimension of the label or the hidden state size of the GRU also afects the performance of the model.
To compare the efect of the label attention network, we conduct experiments with BERT’s token classification model as a comparison. BertForTokenClassification 1 is a fine-tuning method of the BERT model in downstream tasks, and it has obtained an F1 score of 0.924 on the CoNLL2003 named entity recognition data set. It uses BERT as the semantic embedding and then uses the fully connected layer as the output model. We replace the semantic embedding module as the semantic embedding of BertForTokenClassification, to keep the semantic embedding 1https://huggingface.co/transformers/model_doc/bert.html#bertfortokenclassification the same. The comparison between the results our model and of BertForTokenClassification is shown in Table 3.
We also fine-tune the other hyperparameters of the model, and Table 4 shows all the hyperparameters of the model submitted to the final evaluation results. And the F1 score of the result on the test set is 0.719. Our experimental implementation is based on the PyTorch [15] framework and the transformers package [16] is used to get the pre-trained language model. In the data preprocessing stage, the Spacy [17] is used for tokenization and tagging.
We analyze the error of the final submission result, and there are two main errors: the wrong recognition of the entity boundary and the missed recognition of the entity. For the first type of error, as shown in Figure 3, the true annotation is "infiltración de médula ósea ", but the model recognizes two sub-parts "infiltración de " and "ósea". We believe that the model may not be able to learn the semantic dependency information of long entities because the average entity length in the training set is 2.30. Therefore, when recognizing such long entities, the model tends to recognize them as multiple short entities, resulting in the precision of 0.716. Regarding the second error, we believe that the model’s ability to model sentence semantics is insuficient and cannot fully recognize entities. This also results in a low recall, with a final recall of 0.721.
We develop an automated system based on the Cantimist dataset to efectively identify cancer entities in clinical texts. The system consists of three parts: the semantic embedding module, the semantic reasoning module, and the output module. In the semantic embedding module, BEOT, a BERT model trained in Spanish, embeds the text information into a high-dimensional semantic space and obtains the semantic representation of the text. In the semantic reasoning module, a bidirectional GRU that can capture long-distance dependencies is used to model sentences in the forward and backward directions. In the output module, a label-based multihead attention mechanism is used to infer the constraint relationship between labels. We select the best hyperparameters through tuning experiments and obtain good results compared with the baseline in the final evaluation.
However, our model still has a lot of room for improvement. There are some defects in the model’s ability to model sentence semantics, such as some entities missing and entity boundary error. Our future work is mainly to solve the problem of model modeling of sentence semantics. It may be that this data set is a professional text about tumor morphology and BEOT is trained on ordinary corpus. The model we proposed is not able to model sentence semantics well. In the next work, we will try to use graph neural networks to model text semantics to have better results in the recognition of long entities.
This work was supported by the Natural Science Foundations of China under Grant 61463050. [15] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison,
L. Antiga, A. Lerer, Automatic diferentiation in pytorch (2017). [16] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Brew, Huggingface’s transformers: State-of-the-art natural language processing, ArXiv abs/1910.03771 (2019). [17] M. Honnibal, I. Montani, spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing, To appear 7 (2017).