Extracting key concepts from electronic health records makes it easier to access and represent the information contained in them. The growing number of these documents and the complexity of clinical narrative makes it dificult to label them manually. This is why the development of systems capable of automatically extracting key information is of great interest. In this paper, we describe a deep learning architecture for the identification of named tumor morphology entities. The architecture consists of convolutional and bidirectional Long Short-Term Memory layers and a final layer based on Conditional Random Field. The proposed system (HULAT-UC3M) participated in CANTEMIST-NER and obtained a micro-F1 of 83.4%
With the exponential growth of clinical documents, natural language processing (NLP) techniques have become a crucial tool for unlocking critical information to make better clinical decisions. Understanding health-related problems requires the extraction of certain key entities such as diseases, treatments or symptoms and their attributes from textual data.
The identification of specific entities of interest inside medical documents can be addressed as
a Named Entity Recognition (NER) problem. The diferent approaches to solving this problem
range from dictionaries and rule-based systems, machine learning, deep learning, and hybrid
systems. In similar shared tasks, such as eHealth 2019 [
This paper describes a participation of the team HULAT-UC3M in CANTEMIST 2020
challenge [
The core of the proposed system is an adaptation of [
The rest of the paper is organized as follows. In Section 2 we briefly describe the datasets provided for the CANTEMIST task. In Section 3, we describe the architecture of our system. Section 4 presents the results obtained for our system. In Section 5, we provide the conclusions.
The dataset [
We pre-process the text of the clinical cases taking into account diferent steps. First, the texts
are split into tokens and sentences using the Spacy2, an open-source library that provides
1http://brat.nlplab.org/standof.htm
2https://spacy.io/
support for texts in several languages, including Spanish. We have decided to use Spacy, instead
of other more specialized tools for processing clinical texts such as SciSpacy3, due to the fact
that they do not support Spanish texts. Finally, the text and its annotations are transformed
into the CoNLL-2003 format using the BIOES schema [
• S: indicates that an entity is comprised of a single token.
In this section we present the diferent attributes considered to be the input of the deep learning stack.
• Words: A representation based on pre-trained word embeddings has been used with
FastText [
The model implemented, as shown in Figure 1, works on two levels of maximum sequence length. First, a bidirectional LSTM layer receives the input features, the sequence of characters and syllables being previously processed by a convolutional and global max pooling block, for a maximum sequence length of 10. The reshaped output of this layer is concatenated with the input features, but this time for a maximum sequence length size of 50, and serves as the input for a new BiLSTM. This layer connects directly to a fully connected dense layer with ℎ activation function.
The last layer (CRF optimization layer) consists of a conditional random fields layer. This layer receives as input the sequence of probabilities of the previous layer in order to improve predictions. This is due to the ability of the layer to take into account the dependencies between the diferent labels. The output of this layer provides the most probable sequence of labels.
The hyperparameters of our model used are listed below:
• Maximum character sequence length: 30
• Maximum syllable sequence length: 10
• Character convolutional filters : 50 filters of size 3
• Syllables convolutional filters : 50 filters of size 3
• First BiLSTM hidden state dimension: 300 for the forward and backward layers
• Second BiLSTM hidden state dimension: 300 for the forward and backward layers
• Dense layer units: 200
• Dropout: 0.4
• Optimizer: ADAM optimizer [
The system has been developed in python 3 [
In our experiments, we apply the standard measures precision, recall, and micro-averaged F1-score to evaluate the performance of our model. These metrics are also adopted as the evaluation metrics during CANTEMIST task.
We utilized the training set for training the model and exploited the development set to hyperparameter fine tuning. In the prediction stage, we combined the training and development sets to training the model. The detailed hyper-parameter settings are ilustrated in Table 2 ‘Opt.’ denotes optimal.
Considering the optimal hyper-parameters, we use these in our model to process the test set provided by CANTEMIST in subtask CANTEMIST-NER. Our proposal, as the Table 3 shows, reaches a F1-score of 0.834, with an precision of 0.826 and a recall of 0.843. The results show that a proposal based on BiLSTM-CRF can be interesting.
Furethemore, we manually analyzed the erorrs generated by our system on the corpus set after the CANTEMIST-NER task. The main errors can be classified into three categories: • Incorrect boundaries: This type of error is usually caused by including more information, especially related to the location of the tumor, than is necessary. An example of this found in the test set is where our system predicts "nódulo pulmonar contralateral" (contralateral pulmonary nodule) as an entity while the gold standard annotation shows "nódulo pulmonar" (pulmonary nodule). This error is also caused by not including important information such as the stage of the tumor. • Missing the tumor morphology mention (Missed): This error occurs when the system does not recognize a tumor morphology mention in the clinical text. This type of error is often caused when the model confuses terminology that in other documents refers another medical condition, but in the current document refers to tumor morphology. An example of this is "lesión pulmonar" (lung lesion), which depending on the context may or may not refer to tumor morphology. • Incorrectly distinguishing the tumor morphology mention (Incorrectly distinguished): This type of error is often caused when the model confuses terminology that in other documents refers to tumor morphology, but in the current document refers to another medical condition. An example of this is "lesión ósea" (bone lesion), which depending on the context may or may not refer to tumor morphology.
Table 4 details the proportion of errors made by our system in the predictions on the test set in the CANTEMIST-NER task.
By analyzing these error examples, we infer the document-level information may be helpful for our system.
Due to the exponential increase in electronic health records, the need to develop systems capable of automatically identifying and classifying entities in order to facilitate the use of electronic information and resources has become stronger.
The shared task CANTEMIST is focused on the recognition of tumour morphology entities with the aim of contributing to oncological and clinical research and collecting potentially important clinical variables for cancer treatments hidden in medical texts, which are essential to promote health quality improvements and personalised medicine in a more systematised way for the advancement of cancer care.
In this document, we propose a system based on deep learning with bidirectional LSTM and CRF layers for the NER task. Our system achieves a micro-F1 of 83.4%.
In future work, our goal is to explore diferent ways to include document-level information, as well as examine to other deep learning architectures and other types of embeddings such as contextual embeddings.
This work was supported by the Research Program of the Ministry of Economy and Competitiveness - Government of Spain, (DeepEMR project TIN2017-87548-C2-1-R).
The sources for the HULAT-UC3M participation are available via • GitHub https://github.com/ssantamaria94/CANTEMIST-Participation,