Traditional pipeline models for medical named entity recognition and normalization (MER and MEN) suffer from error propagation. To tackle the error propagation problem, we propose a novel joint deep learning method for the 2020 IberLEF shared task on MER and MEN, where MER is regarded as a machine reading comprehension (MRC) problem and MEN as multiple sequence labeling problems corresponding to normalized hierarchical tumor codes. In the 2020 IberLEF shared task, our proposed joint model achieves an F1 score of 0.87 on MER and an F1 score of 0.825 on MEN, and significantly outperforms pipeline models for comparison.
not included in the terminology of this concept. In the 2020 IberLEF shared task, our proposed joint model achieves an F1 score of 0.87 on MER and an F1 score of 0.825 on MEN, and significantly outperforms pipeline models for comparison.
As shown in Figure 2, we develop a joint deep learning model for the 2020 IberLEF shared task on MER and MEN. For the MER task, we adopt a machine reading comprehension model to detect ME spans. For MEN, we regard it as multiple sequence labeling tasks. Each part is presented in the following section in detail. 2.1.
The 2020 IberLEF shared task organizers provide a corpus, called CANTEMIST, including a total of 6233 clinical notes in Spanish, 1301 out of which are manually annotated with “MORFOLOGIA_NEOPLASIA” entities mapped to 8410 normalized turor codes. The annoted corpus is further split into a training set of 501 notes, a development set (dev) of 250 notes, a supplementary development set (sdev) of 250 notes and a test set of 300 notes mixed with other 4932 notes as background (bg). A tumor code consists of six digital chars plus one relevant modifier denoted by “ABCD/EF/H” and can be divided into four parts of different meanings as follows: “ABCD”-tumor/cell type, ‘E’-behavior, ‘F’-differentiation and ‘H’-relevant modifier not included in the terminology of this concept. Table 1 lists the statistics of the corpus in detail. document 2.2.
12 = (17 + 1),
Different from most existing models that regard MER as a sequence labeling problem that needs to tag
each token with entity boundary and type, in this paper, we regard MER as an MRC problem, whose
task is to answer questions regarding different types of entities based on given passages. Following
previous studies [
:2 = (:7 + :), where 7 = (17) ∈ {0,1} represents whether the i-th position is the start position of an entity and 7 = (:2 ) ∈ {0,1} represents whether the i-th position is the end position of an entity, 1 and : are parameter matrices, 1 and : are bias vectors.
During the training phase, we adopt the cross entropy loss to optimize the parameters of our MER model, which is defined as follows: 1EFGE = ∑1 (1, 1), (3) :+- = ∑: (:, :), (4) L:G = 1EFGE + :+-, (5) where CE is the cross entropy loss, 1 and : are the possibilities of predicted start position and end position , 1 and : are the possibilities of gold standard start position and end position . 7FNO- = (FNO-7 + FNO-), 7:P = Q:P7 + :PR, 7ST = QST7 + STR, 7 = 7FFNO-/7:P/7ST, (1) (2) (6) (7) (8) (9) (10) 2.3.
The task of MEN is to map a medical named entity to a normalized code in a given vocabulary. In this paper, we convert MEN into a multiple sequence labeling problem, where each token is labeled with three normalized subcodes as shown in Figure 2, where the behavior code and differentiation code are combined together as we believe they are strongly related to each other. The subcodes of the i-th token in passage can be predicted by equations (6), (7) and (8) defined as follows: where 7FNO-, 7:P and 7ST are the three subcodes, FNO-, :P, ST are parameter matrice and FNO-, :P, and ST are bias vectors..
Similar MER, we adopt the cross entropy loss for model parameter optimization. The loss of MEN is defined as follows: L:+ = V QOWXYZ, OWXYZR + QO[\, O[\R + QO]^, O]^R,
O where O∗ is the possibility of each predicted subcode ∗ and O∗ is the possibility of each gold standard subcode ∗.
The total loss of our joint model is the weighted sum of the MER loss and MEN loss: where is the loss weight. 2.4.
E`EFa = L:G + L:+, (11)
The performances of all models on both MER and MEN task are evaluated by concept-level precision (P), recall (R) and F1-score (F1) under the exact-match criterion.
We first investigate the effect of combination parameter of different values (0.5 vs 1.0) on our
joint model and then compare the joint model with a pipeline method that uses the same method for
MER and a generation model SGM [
The performance of our joint model is listed in Table 2, where the highest P, R, and F1 scores of the model on MER and MEN are highligted in bold. The highest F1 score of MER is 0.87 when the loss weight is set as 0.5, and the highest F1 score of MEN is 0.825 when loss weight is 1.0. In total, our model achieves better performance when = 1.0.
As shown in Table 3, the joint learning method yields a higher F1 score than the pipeline method. This demonstrates the joint model benefits from the shared representation of word representation. For the MER task, the joint learning method outperforms the pipeline method by achieving a 0.868 F1 score. For the MEN task, the joint learning method can bring 4.3% F1 score improvements over the pipeline method. Due to the high imbalance of codes, there are ubiquitous code 8000/6. When removing 8000/6 mentions (denoted as No-Metastasis), the joint learning method has a slight change on P, R and F1 score, which indicates the robustness of the joint model. joint
Though our joint learning method shows a great improvement over the pipeline method, there are still some errors on the MER task. 1) Long tail problem is the main obstacle. The number of the entity mentions containing over 10 words is about 60, but our model can only recognize 10 of them. 2) Nested entity mentions are difficult to recognize. Though our model tries to recognize the nested entities, it is difficult for our model to recognize them because of its rareness in the training set. 3) If a text has “A y (and) B”, our model finds it difficult to judge whether A and B are both entities.
The joint learning method shows a great improvement in the MEN task, but there are some limits. The number of sequence labeling submodels has a huge impact on the results. When we further separate the behavior part and the differentiation part of tumor code, the MEN F1 score on the development set decreases from 0.794 to 0.708, indicating that the behavior and differentiation have a strong relationship. In the future, we plan to explore how to detect the relationships among different parts of tumor code automatically in the model.
In this study, we propose a joint learning method for medical named entity recognition and medical named entity normalization. We utilize a machine reading comprehension model to solve thee MER task and a multiple sequence labeling model to solve the MEN task. Experimental results show the effectiveness of our model.
This paper is supported in part by grants: National Natural Science Foundations of China (U1813215, 61876052 and 61573118), Special Foundation for Technology Research Program of Guangdong Province (2015B010131010), National Natural Science Foundations of Guangdong, China (2019A1515011158), Guangdong Province Covid-19 Pandemic Control Research Fund (2020KZDZX1222), Strategic Emerging Industry Development Special Funds of Shenzhen (JCYJ20180306172232154 and JCYJ20170307150528934) and Innovation Fund of Harbin Institute of Technology (HIT.NSRIF.2017052).
Gonzalez-Agirre A, Marimon M, Intxaurrondo A, Rabal O, Villegas M, Krallinger M. PharmaCoNER: Pharmacological Substances, Compounds and proteins Named Entity Recognition track. In: Proceedings of The 5th Workshop on BioNLP Open Shared Tasks. Association for Computational Linguistics; 2019:1–10. doi:10.18653/v1/D19-5701 Marimon M, Gonzalez-Agirre A, Intxaurrondo A, et al. Automatic De-identification of Medical Texts in Spanish: the MEDDOCAN Track, Corpus, Guidelines, Methods and Evaluation of Results. In: Proceedings of the Iberian Languages Evaluation Forum Co-Located with 35th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2019, Bilbao, Spain, September 24th, 2019. ; 2019:618–638. http://ceur-ws.org/Vol2421/MEDDOCAN_overview.pdf Piad-Morffis A, Gutiérrez Y, Consuegra-Ayala JP, et al. Overview of the eHealth Knowledge Discovery Challenge at IberLEF 2019. In: Proceedings of the Iberian Languages Evaluation Miranda-Escalada A, Farré E, Krallinger M. Named entity recognition, concept normalization and clinical coding: Overview of the CANTEMIST track for cancer text mining in Spanish, Corpus, Guidelines, Methods and Results. In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020). CEUR Workshop Proceedings. ; 2020.
Xiong Y, Shen Y, Huang Y, et al. A Deep Learning-Based System for PharmaCoNER. In: Proceedings of The 5th Workshop on BioNLP Open Shared Tasks. Association for Computational Linguistics; 2019:33–37. doi:10.18653/v1/D19-5706 Leaman R, Lu Z. TaggerOne: joint named entity recognition and normalization with semi-Markov Models. Bioinform. 2016;32(18):2839–2846. doi:10.1093/bioinformatics/btw343 Lou Y, Zhang Y, Qian T, Li F, Xiong S, Ji D. A transition-based joint model for disease named entity recognition and normalization. Bioinform. 2017;33(15):2363–2371. doi:10.1093/bioinformatics/btx172 Zhao S, Liu T, Zhao S, Wang F. A Neural Multi-Task Learning Framework to Jointly Model Medical Named Entity Recognition and Normalization. In: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019. ; 2019:817–824. doi:10.1609/aaai.v33i01.3301817 Li X, Feng J, Meng Y, Han Q, Wu F, Li J. A Unified MRC Framework for Named Entity Recognition. arXiv preprint arXiv:191011476. Published online 2019.