This paper describes the development of a Named Entity Recognition (NER) system to automatically detect tumor morphology mentions in medical documents, known as ICD-O codes, International Classiifcation of Diseases for Oncology. This study is developed as part of the Cantemist program of the Plan of Advancement of Language Technologies (Plan TL). This work tries to contribute to the existing NER technologies that focus on Spanish health-related documents. This is a necessary task given the amount of research, regarding the health sector, that is written in Spanish and the benefits it could bring to the medical environment. In fact, since most NER techniques are developed for English, this research cannot be completely exploited. In this research, we explore diferent machine learning techniques such as CRF, a Bidirectional Long short-term memory (Bi-LSTM) and a Bidirectional Encoder Representations from Transformers (BERT) to address the task of detecting tumor morphology mentions from clinical texts written in Spanish.
Natural Language Processing (NLP) has become vital since the amount of information to which
people have access nowadays cannot be easily managed. To solve this, NLP ofers tools that vary
based on the intention of the user, such as translating, summarizing or extracting information
from a text. [
Focusing on NER, there are currently many technologies that achieve state-of-the-art
performance. However, there are two key aspects that force to keep developing in this field. On the
one hand, the performance of NER systems is dependent on the domain. As a result, this makes
it necessary to construct domain specific NER systems. In fact, this task becomes crucial in the
medical domain given the amount of research in this field and the advantages it could bring to
patient diagnosis, prognosis and further research. [
On the other hand, despite the generalization that some NER systems can achieve, the performance of the systems is also dependent on the language it was built for. Therefore, this makes it necessary to build systems specific for the diferent languages. For instance, focusing on Spanish, there is a huge amount of biomedical research written in this language. However, since most NER techniques are developed for English, this research cannot be completely exploited.
Disease recognition and normalization in medical texts is a challenging task given the
variability and complexity of the disease mentions. Currently, the main technique followed in this
context is the clinical coding, which consists in the collection of codes regarding the taxonomy
of a disease, signs, symptoms and medical procedures. These are standard codes provided by
the ICD10, International Classification of Diseases (CIE10 in Spanish). In particular, the ICD-O
codes (CIE-O-3.1 in Spanish), International Classification of Diseases for Oncology, allows to
code tumor morphology mentions in health-related documents written in Spanish. [
Under this context, this study proposes a NER system to automatically detect tumor
morphology mentions in a medical document. It is developed as part of the Cantemist track [
We focus on the NER task. This way, it will aim at constructing a system to identify tumor morphology mentions in the corpus provided by the Cantemist organizers and contribute to the existing NER technologies and resources that focus on Spanish health-related documents. Our study explores three diferent machine learning approaches: 1) CRF classifier as baseline, 2) Bidirectional Long short-term memory (Bi-LSTM) and 3) Bidirectional Encoder Representations from Transformers (BERT) to address the task of detecting tumor morphology mentions from clinical texts written in Spanish.
This study will be focusing on domain specific NEs, more specifically, on a unique entity type: “Morfología Neoplasia”.
NER aims to identify and categorize words or expressions inside a text that represent entities.
NER systems must confront many challenges such as the construction of a generic entity tagger
given the variability of the entity set among diferent domains, the diversity of languages and
the ambiguity within each of them [
Two key aspects in NER systems are to employ context information in the identification of
entities and assume inter-dependency between the words in the text. [
Since entities can be words or phrases, they are usually labeled following the IOB format,
where every token that conforms an entity is assigned a tag ‘I’, ‘O’ or ‘B’ based on its location,
followed by the entity type. In fact, the IOB format stands for “Inside, Outside, Beginning”.
Therefore, if a token is at the beginning of an entity, it will be labeled as “B”, as “I” if the token
is inside an entity and as “O” if the token is not part of an entity. [
Along the years, several approaches have been employed to address the NER task. The main approaches include:
Rule-based It is based on hand-crafted rules, that can be either semantic or syntactic. This
approach does not require annotated data and is highly dependent on the domain and can rely
on dictionaries. It usually provides high precision but with a low recall.[
Common supervised methods employed in NER are Hidden Markov Models, Decision Trees,
Maximum Entropy Models, Support Vector Machines and Conditional Random Fields. [
Deep learning The use of deep learning in NER became more common in recent years. As
an advantage to previous techniques, the extraction of features is learned automatically by the
model, without complex feature engineering. This is done by stacking various layers in the
neural network, where diferent abstractions of the data are obtained. Another advantage of
deep learning is the use of non-linear activation functions that allow to learn more complex
features. [
As previously mentioned, NER is a challenging task for various reasons such as the variability
of the entity sets among diferent domains and the dependency on the quality of the annotations.
However, when it comes to the biomedical domain, the NER task becomes even more challenging
given the domain specific terminology, the use of acronyms or abbreviations and non-standard
terms. Furthermore, Bio-NER systems are afected by the lack of comprehensive biomedical
entities dictionaries. [
Many studies have shown that the combination of a Bidirectional Long short-term memory
(BiLSTM) followed by a CRF achieves state-of-the-art performance in NER [
Wei et al. [
Moreover, Choo and Lee [
The Cantemist corpus [
In the BRAT format, each line represents an entity mention. Each entity annotation includes
an ID (‘T1’, ‘T2’. . . ) where ‘T’ stands for “text bound annotation”, followed by the entity type
(‘MORFOLOGIA NEOPLASIA’), start-end ofsets and the text of the annotation. The start-end
ofset indicates the position within the file of the first and last character in the entity. [
Entities can be formed by several tokens (words). In fact, the average number of words that
conform each entity is 2. However, the BRAT format in which the annotations are given show
continuous text-bound annotations, where only the start ofset of the first word and the end
ofset of the last word that form the entity are given (see Figure 2). That is why, a pre-processing
stage that involves converting such annotations into discontinuous text-bound annotations has
been applied (see Figure 3). [
The IOB format is mostly used to tag each token. However, after analyzing the Cantemist
corpus, the presence of nested entities, which are entities embedded in another entity, was
noticed (see Figure 4). On the contrary, those entities that do not appear in various annotations
are known as flat entities. To solve this problem, we have used an extension of the IOB format,
the BIOES-V format. [
Regarding the training dataset, over which the following methods will be trained, the distribution of tokens belonging to each BIOES-V tag is presented in Table 1. Table 1 also includes the distribution of tokens of the test set. Figure 5 shows the main preprocessing tasks involved in our study.
As baseline, we propose the use of a CRF classifier, a type of probabilistic discriminative model which aims to predict an output variable while giving huge relevance to the sequence of predictions. The main diference with generative models is that these rely on strict dependency assumptions, whilst this discriminative model relaxed such assumptions. This way, it can employ features where it exists a dependency. Until the implementation of deep learning methods, CRF provided the best performance in NER tasks among machine learning methods.
CRF models the conditional distribution p(y|x), which is the probability of label sequences
given observation sequences and follows the formula below. [
1
( )
=1 =1
( | ) =
exp (∑
∑ ( −1, , )),
normalization factor defined as follows. [
Here, represents the parameters of the distribution and is the function that defines the sequences’ labels. is the ( ) = ∑ exp (∑
∑ ( −1, , )),
=1 =1
The training of the model involves working with sentences as the input sequence. In addition, for every token in a sequence, the model needs as parameters, the position of the current token, its PoS (Part of Speech) tag, the BIOES-V label of the current token and the label of the previous token. This way, context information is also being considered. [14]
We also implement a BiLSTM (Long Short-Term Memory) network followed by a CRF classifier. On the one hand, the LSTM is a type of RNN (Recurrent Neural Network). The main aspect of these RNNs is that they employ context information, more specifically, information from past observations, to make predictions over the current observation. [15] In this study, the observations are tokens from the text that are used as sequential inputs to the network. This ability to “remember” information is crucial in NER since this is a sequence labelling problem where there is interdependency between the tokens in the text. However, RNNs sufer from vanishing gradient problem caused by the increasing amount of information to be “remembered”. To solve this, LSTMs difer from common RNNs by “forgetting” information and “making space” for more important one. [15] In addition, the implementation of a bidirectional LSTM becomes more appropriate when working with NER since it allows not only to remember information from past observations but also to consider future information. This is achieved by employing two independent LSTM processes working in opposite directions. As a result, the final layer (a CRF classifier) in the network receives two vectors, the outputs of the two LSTMs layers, as an input. Then, the CRF considers the label predicted for the previous observation in order to predict the label of the current one. This behavior is necessary in an NER task given the interdependency between tokens. We have implemented three diferent approaches based on BiLSTM-CRF.
a word vocabulary from the words present in the training dataset and map each word to a
numeric vector of 40 components. It must be mentioned that the length of each sentence is
being set to a fixed size, 75. Then, it assigns a random vector for each token in the vocabulary.
Unfortunately, it does not capture relationships or similarities between tokens. In addition, it
may occur that certain tokens in the test dataset do not appear in the training dataset. Therefore,
regardless of the similarity of these tokens to others in the vocabulary, these tokens will be
mapped to the value that identifies the “Unknown” label. [15]
Approach 2: BiLSTM-CRF with a pre-trained word embedding in Spanish Word
embedding is a technique that consists on representing words in a vocabulary as vectors of real
numbers while capturing semantic and syntactic information. [
Approach 3: BiLSTM-CRF with character embedding This technique implements the pre-trained word embedding explained in approach 2. As a novelty from the previous approach, it represents each character in the words as a vector of a fixed dimension. For this, it performs character embedding as a previous layer to represent words as characters. This approach is useful when working with out of vocabulary words (OOV), which are words that do not appear in the vocabulary that the model was trained with. In sum, this approach combines two feature representations of the text as inputs to the system. On the one hand, every token in the text is encoded into a 300-element vector using a vocabulary from a pre-trained word embedding. On the other hand, every character that conforms a token is encoded using a character vocabulary also learned from the pre-trained word embedding. This way, this approach considers two fixed parameters; the maximum sequence length, fixed to 75, and the maximum word length, fixed to 10.
Finally, we also explore the use of Bidirectional Encoder Representations from Transformers
(BERT) [
Furthermore, the process of BERT consists of two stages. The first stage is the model
pretraining over unlabeled data and the second stage is approached either by performing feature
based or fine-tuning tasks. [
Since this task involves working with clinical cases in Spanish, a multilingual cased version
of BERT is employed instead [21]. It has fixed model sizes: L=12, H=768, A=12, and Total
Parameters=110M. Here, L denotes the number of layers (or transformer blocks), H is the hidden
size and A is the number of self-attention heads. This way, the only parameters to focus on
while performing fine-tuning are maximum length (the context or sequence length to consider),
batch size, learning rate and number of epochs. [
In sum, we use an already pre-trained cased text BERT model. Therefore, no preprocessing steps has been performed over the corpus. Regarding the method’s parameters, the maximum sequence length has been fixed to 75, as in previous approaches. When it comes to the training of the model, a batch size of 32, a learning rate of 3e-5 and 3 epochs have been employed. It must be mentioned that the number of epochs specify the number of times the model goes through all the data. Therefore, in order to avoid overfitting, this value must not be too large.
The performance of the models has been evaluated using several metrics. The primary metrics employed are precision, recall and f1 score. What’s more, it will also be employing micro-average scores instead of macro-average scores since the number of tokens belonging to each entity type is imbalanced. The main diference between micro-average and macro-average scores is that micro average scores are computed jointly for all the entity types whilst the macro average scores involve computing the metrics individually for each entity type and then combining them by performing the mean.
The following methods have been trained using the training dataset and fine-tuned employing the labeled development datasets. Then, once the optimal parameter combination had been achieved, the final model has been trained employing the training and development datasets.
First, we will present and discuss the performance of our approaches by considering their micro-average results calculated over the BIOES-V tags used to represent our tokens. Finally, we also present the general results provided by the Cantemist organizers.
First, we propose the CRF classifier, which has shown good performance in literature [
BiLSTM-CRF with random initialization achieves a micro average F1 score of 0.71 and 0.73 on the development datasets 1 and 2 respectively, when analyzing its performance in identifying tokens that belong to an entity. Regarding the test dataset, it has achieved a micro average F1 score of 0.78 as seen in Figure 7.
BiLSTM-CRF with a pre-trained word embedding in Spanish shows a micro average F1 score of 0.75 and 0.78 on the training and development datasets 1 and 2 respectively, when analyzing its performance in identifying tokens that belong to an entity. On the other hand, an f1 score of 0.81 has been obtained over the test dataset as seen in Figure 8.
BiLSTM-CRF with character embedding obtains a micro average F1 score of 0.75, 0.72 and 0.76 on the training and development datasets 1 and 2 respectively, when analyzing its performance in identifying tokens that belong to an entity. Regarding the test dataset, it has obtained an F1 score of 0.78 as seen in Figure 9.
Finally, BERT achieves a micro average F1 score of 0.78 and 0.80 on the development datasets 1 and 2 respectively, when analyzing its performance in identifying tokens that belong to an entity. Focusing on the test set, it achieved a micro average f1 score of 0.82.
In short, CRF established a baseline with a micro average f1 score of 0.78. Then, focusing on the deep learning methods, three strategies that implement a Bidirectional LSTM combined with CRF have been approached. The first strategy consisted on a random initialization of the vectors in the vocabulary and did not succeed in defeating the baseline method (see Figure 11). The second BiLSTM-CRF strategy involved a pre-trained word embedding in Spanish for token representation. It achieved a better performance than the first BILSTM-CRF approach and baseline approach, CRF, (see Figure 11). The third BiLSTM-CRF strategy consisted on incorporating a character embedding along with the token representation did not achieve a better performance than the second approach of BiLSTM-CRF nor the baseline approach (see Figure 11). Finally, the last deep learning approach to be assessed was BERT. This method achieved to defeat both the baseline method (CRF) and the previous Bi-LSTM approaches explored (see Figure 11). Therefore, we can conclude that BERT is the best option to recognize tumor morphology mentions in clinical texts.
However, the performance of the models must also be assessed based on the correctly identified entities in each clinical case. This way, this assessment does not consider if a token has been correctly identified as part of an entity, but if an entity, which can be formed by several tokens, has been identified in its complete form. The results of this evaluation have been captured in Table 2. It shows that the second approach of BiLSTM-CRF initialized with a pre-trained word embedding model has the best performance among the diferent methods explored.
This study has focused on developing a NER system to automatically identify tumor morphology entity mentions in health-related documents in Spanish. As previously mentioned, it has been developed under the Cantemist track, from which the corpus employed has been obtained.
For this purpose, we have explored diferent machine learning approaches such as CRF, Bi-LSTM and BERT. Although the approaches show very similar performance, we can conclude that Bi-LSTM with pre-trained word embeddings shows the top F1 (0.797). However, if we study the micro-average F1 calculated over the BIOES-V format to represent our tokens, we see that BERT provides better results (micro F1=0.82) than the other approaches.
As future work, we plan to extend our deep learning models by incorporating semantic information from biomedical dictionaries (such as entity embeddings). We will also explore other hybrid deep learning architecture.
This work was supported by the Research Program of the Ministry of Economy and Competitiveness - Government of Spain, (DeepEMR project TIN2017-87548-C2-1-R) and the Interdisciplinary Projects Program for Young Researchers at Universidad Carlos III of Madrid founded by the Community of Madrid (NLP4Rare-CM-UC3M). . [13] R. M. R. Zavala, P. Martínez, I. Segura-Bedmar, A hybrid bi-lstm-crf model for knowledge recognition from ehealth documents., in: TASS@ SEPLN, 2018, pp. 65–70. [14] N. Patil, A. Patil, B. Pawar, Named entity recognition using conditional random fields,
Procedia Computer Science 167 (2020) 1181–1188. [15] X. Ma, E. Hovy, End-to-end sequence labeling via bi-directional lstm-cnns-crf, arXiv preprint arXiv:1603.01354 (2016). [16] Y. Goldberg, O. Levy, word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method, arXiv preprint arXiv:1402.3722 (2014). [17] J. Pennington, R. Socher, C. D. Manning, Glove: Global vectors for word representation, in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543. [18] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information, Transactions of the Association for Computational Linguistics 5 (2017) 135–146. [19] F. Soares, M. Villegas, A. Gonzalez-Agirre, M. Krallinger, J. Armengol-Estapé, Medical word embeddings for spanish: Development and evaluation, in: Proceedings of the 2nd Clinical Natural Language Processing Workshop, 2019, pp. 124–133. [20] C. Cardellino, Spanish billion words corpus and embeddings (march 2016), URL http://crscardellino. me/SBWCE (2016). [21] J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, J. Pérez, Spanish pre-trained bert model and evaluation data, in: Practical ML for Developing Countries Workshop@ ICLR 2020, 2020.