This paper describes a two-stage system to solve tumor entity detection and coding in Spanish health records. This system is submitted to the CANcer TExt Mining Shared Task (CANTEMIST), a challenge in the IberLEF 2020 Workshop. We include a comparison between two kinds of systems to tackle this problem. The first kind employ feature-based Conditional Random Fields (CRF), and the second kind is based on deep learning models. The reported experiments show that our proposals and their combination achieve a micro-F1 of 83.1% and 78.6% on the test data set for the first and second sub-tasks, respectively, and a MAP of 79.7% on the third sub-task.
task requiring all tumor morphology mentions being returned with their corresponding eCIE-O3.1 codes; and 3) CANTEMIST-CODING track, which requires returning, for each of document, a ranked list of its corresponding ICD-O-3 codes [2] (Spanish version: eCIE-O-3.1[3]).
As participants in the contest, we tackled the problem by designing several systems based on Conditional Random Fields (CRFs) and deep learning models. We also proposed a combination of these systems to improve the results.
For the first task (NER task), we design two systems (1-BiLSTM and 2-BiLSTM) based on Artificial Neural Networks (ANNs) with Bi-directional Long Short-Term Memory neural networks (Bi-LSTM), and a third one relying on feature-based Conditional Random Fields (CRF). Then, we combined the results of the three systems using voting.
For the second and third tasks (normalization and coding tasks), we have designed a system based on Convolution Neural Networks (CNNs) and Long Short-Term Memory neural networks (LSTMS).
The remainder of the paper is organized as follows. In Section 2, we briefly describe the data. Section 3 describes the systems we propose. Results and discussions are presented in Section 4. Section 5 presents the conclusions and depicts some lines of future work.
The proposed systems are trained and validated on train and development datasets provided by the organizers of the contest. The corpora released for the tasks consists of 1,300 Spanish medical records, divided into 500 as training data, 250 for each dev1 and dev2 data, and 300 as test data. Table 1 shows general statistics about the four datasets.
We developed several systems to detect and code tumor named entities in Spanish health records. The next subsections describe the steps we followed to train and use these systems. Figure 1 describes the general data flow of the system.
In this step we performed a sentence splitting and text tokenization of the health records. For this, we used the spaCy pre-trained model for the Spanish language [4]. Note 1 Note 2
After we got all the tokens from the previous step, we used word embedding to map these tokens to meaningful n-dimensional vectors. The word embedding model has been generated from Spanish medical corpora [5]. The model was built with fasttext and the dataset used to generate the model consisted of two data sources: (i) the SciELO database, which contains full-text articles primarily in English, Spanish and Portuguese, and (ii) a subset of the Wikipedia, which we call Wikipedia Health, consisting on the articles under the following categories: Pharmacology, Pharmacy, Medicine and Biology.
In addition to the available sentences in the training dataset, we increased the number of sentences with named entities by using the list of tumor entities in the file "valid-codes.txt" from the extra resources provided by the organizers. This file contains 4,203 records of tumor named entities with their corresponding eCIE-O-3.1 codes. This data augmentation was performed by choosing a random sentence containing NEs from the training dataset and replacing the existing entity by a random entity from the list of entities in the "valid-code.txt" file.
Input Word Embedding L S T M L S T M L S T M M T S L M T S
L Backward
Layer
This section describes the proposed systems for handling the tumor named entity recognition. The first subsection explains the feature-based NER using CRF, while the second subsection explains the deep learning model using Bi-LSTM neural networks.
In principle, we included this system only for comparison purposes. However, its results were comparable to those of the deep learning model, so we took advantage of that to improve the ifnal results by combining the outcomes of the two systems.
This system was implemented using Python 3.7 with the sklearn-crfsuite package [6]. The input for this system was the tokenized sentences from the text tokenizer. The system extracts some features for every word token like the Part-of-Speach tag (POS), word prefix, word sufix, word length, etc. This is similar to what we did in our previous work in the MEDDOCAN competition [7, 8, 9]. Similarly, we also used the BIO tagging scheme to set the labels of the tokens [10]. As a result, each word token in the medical record is labeled using one of three possible tags: B, I, or O, which indicate if the word is at the beginning, middle, or outside of a Tumor entity.
As shown in Figure 1, the first stage in our system is the NER, which takes as input the sequence of tokens of a sentence and predicts labels using BIO tags. We choose to use Bi-LSTM as our encoder because of its ability to take all the information sequentially and pass it to the classifier. After that, we have a CRF classifier. CRF does sequence labeling for every token considering the label of neighboring tokens.
These kind of systems take as input a list of tokens with a fixed size. In our case, we choose the maximum size of 80 tokens per sentence; extra tokens were truncated as a separated sentence. If the sentence’s length is less than 80, we added pre-defined padding to reach the maximum size.
As introduced before, by means of word embedding, each token is mapped to its corresponding word embedding vector. After that, all vectors are passed to the LSTM layers and, finally, the output of the LSTM is passed to the CRF layer to do the classification. The CRF layer classifies every token to one of the BIO tags.
The second and the third tasks in the competition focus on tumor named entity normalization, where every tumor NE should be mapped to their corresponding CIE-O-3.1 codes. In the following section, we describe our proposed system for these tasks.
The second and third sub-tasks in the competition (Norm and coding tasks) are clinical concept normalization. The input is a list of entities that are detected by the NER system, which should be mapped to the corresponding eCIE-O-3.1 codes. eCIE-O-3.1 is the Spanish version of the International Classification of Diseases for Oncology (ICD-O). This ontology aims to standardize the tumor named entities in health records and make them understandable internationally.
All the codes in the eCIE-O-3.1 come in the form of three codes separated by /. The regular expression for these codes is dddd/dd?(/H)?. The first idea that comes to the mind is to build a system that takes the NER output as input and predicts the corresponding code. However, implementing this system gave us bad results. After we analyzed why this straight forward system did not perform well, our findings were: 1) the three parts of the code vary between diferent tumor name entities; and 2) some of these combinations appear few times in the . . . training dataset, which produces too many combinations. Because of that, we decided to treat these three parts separately, i.e. build a separate classifier for every part.
Our system contains three identical sub-systems as we mentioned before (see Figure 3 and Figure 4). The input for these three models is a list of 20 tokens (because the number of tokens in the tumor named entities is between 1 and 20). We used CNNs because these kinds of networks reduce the computation by exploiting local correlation of the input data. In our case, the tumor named entities contain in average three words, so we used CNNs with 512 filters of the size three. We do 2-max pooling to get the max two numbers for every filter response, which gives us two vectors of size 512. These two vectors were passed to the LSTM to combine them. Finally, the output was passed to the classifier, which is a fully connected layer.
We have proposed four systems. Table 2 provides details about the training and evaluation of each of them. NER models were trained by feeding all the sentences without exception, while Norm models were trained only on those sentences that contain NEs.
The standard metrics to evaluate the systems participating in this competition are the F1-score, for both NER and Norm sub-tasks, and the Mean Average Precision MAP, for the coding subtask. Precision and Recall also are included to give more insights about the performance of the sub-models. Word Embedding ...
Filter 0 Filter 1 Filter 2 Filter 511 ...
Baseline 1-BILSTM 2-BILSTM
CRF CRF+BILSTM 80,7 82,4 80,6 84,4 83,0 82,4 77,6 81,8
F1 P 29.1 18.0 Our systems 81,8 76,5 82,4 77,9 79,1 77,5 83,1 79,8
Norm task
R 73.0 because of the code "8000/6", which appears more than the other codes. Combining the three systems helped to improve the wrongly classified samples for that code but, at the same time, caused several wrongly classified samples for the other codes.
In this paper we describe the implementation of several systems to solve the problem of tumor named entity recognition and normalization. We compared the performance of our systems with a feature-based system. Results show that a combination of several of our systems provided the best results in most cases.
As future work, we plan to use transformer-based models like BERT [11] or XLNet [12], which they are the state of the art in the NLP field nowadays. We expect these models will give us better results on this task.
Partial support to this work has been received from the European Commission (project H2020871042 “SoBigData++”), the Government of Catalonia (ICREA Acadèmia Prize to J.DomingoFerrer and grant 2017 SGR 705), the Spanish Government (projects RTI2018-095094-B-C21 “Consent” and TIN2016-80250-R “Sec-MCloud”) and the Norwegian Research Council (project no. 308904 “CLEANUP”). The authors are with the UNESCO Chair in Data Privacy, but the views in this paper are their own and are not necessarily shared by UNESCO. [1] A. Miranda-Escalada, E. Farré, M. Krallinger, Named entity recognition, concept normalization and clinical coding: Overview of the cantemist track for cancer text mining in spanish, corpus, guidelines, methods and results, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020), CEUR Workshop Proceedings, 2020. [2] Steward/Custodian, International classification of diseases for oncology, 3rd edition (icd-o-3), https://www.who.int/classifications/icd/adaptations/oncology/en/, (accessed: 17.08.2020). [3] Portal estadístico del ministerio de sanidad servicios sociales e igualdad, tabla de códigos de morfología de las neoplasias (cie-o-3.1) válidos, https: //datosabiertos.castillalamancha.es/dataset/registro-de-actividad-de-atenci%C3% B3n-sanitaria-especializada-de-castilla-la-mancha-rae-clm-11, (accessed: 17.08.2020). [4] M. Honnibal, I. Montani, spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing, To appear 7 (2017). [5] F. Soares, M. Villegas, A. Gonzalez-Agirre, M. Krallinger, J. Armengol-Estapé, Medical word embeddings for spanish: Development and evaluation, in: Proceedings of the 2nd Clinical Natural Language Processing Workshop, 2019, pp. 124–133. [6] M. Korobov, sklearn-crfsuite, https://sklearn-crfsuite.readthedocs.io/en/latest/tutorial.html, (accessed: 17.08.2020). [7] M. Marimon, A. Gonzalez-Agirre, A. Intxaurrondo, H. Rodriguez, J. L. Martin, M. Villegas, M. Krallinger, Automatic de-identification of medical texts in spanish: the meddocan track, corpus, guidelines, methods and evaluation of results., in: IberLEF@ SEPLN, 2019, pp. 618–638. [8] F. Hassan, M. Jabreel, N. Maaroof, D. Sánchez, J. Domingo-Ferrer, A. Moreno, Recrf:
Spanish medical document anonymization using automatically-crafted rules and crf., 2019. [9] M. Jabreel, F. Hassan, D. Sánchez, J. Domingo-Ferrer, A. Moreno, E2ej: Anonymization of spanish medical records using end-to-end joint neural networks., 2019. [10] E. F. Sang, J. Veenstra, Representing text chunks, arXiv preprint cs/9907006 (1999). [11] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [12] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, Q. V. Le, Xlnet: Generalized autoregressive pretraining for language understanding, in: Advances in neural information processing systems, 2019, pp. 5753–5763.