Currently, the number of electronic health documents is increasing exponentially. Due to this, there is a growing interest in developing automatic systems to extract interesting information from these texts. In this paper, we describe a deep learning architecture for the identi cation and classi cation of named entities of interest in health documents. The architecture consists of two bidirectional Long ShortTerm Memory layers and a nal layer based on Conditional Random Fields. Our system (Hulat-TaskAB) participated in the ehealthkd-2019 sub-task A and obtained a micro-F1 of 76.63%.
Electronic health records (EHRs) are one of the richest resources for epidemiological studies in order to plan and evaluate strategies to identify and prevent diseases, among others. However, researchers have to manually review a large amount of information, which is a very costly and time-consuming task. Therefore, providing systems capable of automatically identifying and classifying entities or key phrases of interest in these records conforms a vital task.
The identi cation of speci c entities of interest inside medical documents can be addressed as a Named Entity Recognition (NER) problem. This problem has been widely studied and the approaches normally used can be divided into several main categories: dictionaries and rule-based systems, machine learning, deep learning, and hybrid systems.
Dictionary-based methods are limited by the size of the dictionaries
themselves, in addition to the constant growth of vocabulary and spelling errors.
Rulebased approaches usually provide high precision, however, they do not usually
contemplate all existing cases as a result of the complexity of the language. For
example, in [
Furthermore, rule-based and machine learning methods require a previous
generation of syntactic and semantic features, as well as domain-speci c
information. For example, in [
Approaches based on deep learning methods automatically learn relevant
patterns, allowing a certain grade of independence of language and domain.
Moreover, these approaches have been shown to achieve better results than the
best hybrid systems in i2b2 tasks. The system described in [
Considering the above, in this paper, we propose the use of an adaptation
of the NeuroNer tool [
The rest of the paper is organized as follows. In Section 2 we brie y describe the datasets provided for the eHealthkd-2019 task. In Section 3, we describe the architecture of our system. Section 4 presents the results obtained for our system. In Section 5, we provide the conclusions. 2
The data set consists of the annotated corpus of Spanish electronic health documents proposed in the ehealthkd-2019 sub-task A1. For this task, all the documents are provided in BRAT format2, a stando format where the di erent annotations are stored separately from the original text in a similar way to the BioNLP Shared Task stando format3.
As we can observe in Table 1, the training set is composed of 600 sentences annotated manually and contain a total of 4350 key phrases distributed among four classes of di erent categories. The development set is composed by 100 sentences annotated manually with a total of 4350 key phrases. The four categories are listed below: 1 (https://github.com/knowledge-learning/ehealthkd-2019/tree/master/data) 2 http://brat.nlplab.org/stando .html 3 http://2011.bionlp-st.org/home/ le-formats { Concept: Key phrase that indicates a relevant term or idea in the sentence. { Action: Indicates a process or modi cation of concepts. { Predicate: Represents a function or lter in a set of elements. { Reference: Element that refers to a not explicit concept.
In addition, as we can see in Table 2, the di erent categories are represented unbalanced in a similar proportion for both sets.
We pre-process the text of the clinical cases taking into account di erent steps. First, the texts are split into tokens and sentences using the Spacy4, an opensource library that provides support for texts in several languages, including Spanish.
Finally, the text and its annotations are transformed into the CoNLL-2003
format5 using the BIOES schema [
3.2
Bidirectional LSTMs are a type of recurrent neural network (RNN) where the
context of words in the sentence is captured using information from previous
words and information from subsequent words. In addition, to improve the
accuracy of the predictions provided by the BiLSTM layer, the CRF layer uses
information from neighboring (sentence level) tags to predict current tags. Due
to the performance of this type of architectures in entity recognition tasks, we
propose the use of the NeuroNer [
The rst layer (Character-enhanced and token embedding layer) aims to
generate vector representations of the tokens that conform the input sequences.
The direct representation of token to vector (word embedding) can be pre-trained
or can be learned in conjunction with the rest of the model by adjusting its
weights. Pre-trained models can be obtained from a large amount of unlabeled
data with methods such as word2vec, FastText or GloVe [
The character embedding sequence of each token is passed as input to a BiLSTM to obtain character-based word embedding as output. Finally, the representation of word embeddings and character-based word embedding are concatenated for each token, which will be the input for the second BiLSTM layer (BiLSTM prediction layer). This BiLSTM layer aims to obtain the sequence of probabilities for each token to pertain to a given label using the BIOES coding. The label for each token will be the one with the highest probability.
The last layer (CRF optimization layer) consists of a conditional random elds layer. This layer receives as input the sequence of probabilities of the previous layer in order to improve predictions. This is due to the ability of the layer to take into account the dependencies between the di erent labels. The output of this layer provides the most probable sequence of labels.
The hyperparameters of our model used for the eHealthkd-2019 subtask A are listed below: { Word Embeddings: randomly initialized and adjusted during training.
The dimension of the vectors is 200.
{ Character Embeddings dimension: randomly initialized and adjusted
during training. The dimension of the vectors is 25.
{ First BiLSTM hidden state dimension: 25 for the forward and backward
layers
{ Second BiLSTM hidden state dimension: 200 for the forward and
backward layers
{ Optimizer: ADAM optimizer [
A more detailed description of the evaluation can be found on the website6.
To evaluate the trained models, as well as their hyperparameters, we performed a set of experiments with the development dataset provided by the eHealthkd-2019 organizers. We used grid search to adjust the word embeddings dimension, the number of units in the BiLSTM hidden layer, the optimizer and the learning rate.
We can observe in Table 3 our best results on the development set. The run0 model was trained using the hyperparameters mentioned in section 3.2. The run1 model was trained using ADAM as optimizer, with a word embeddings dimension of 200 and 100 units in the second layer of the BiLSTM. The run2 model was trained using ADAM employing a word embeddings dimension of 100 and 100 units in the second layer of the BiLSTM.
Considering that the model run0 achieved the best results, this model was used to process the test set provided by eHealthkd-2019 in subtask A. The results obtained can be seen in Table 4.
As we see in Table 4, the test set of subtask A contains 646 key phrases. In total, the spans and labels match correctly in 476 of them and partially in 58. One of the elements that most a ect our result is the number of spurious phrases, resulting in decreased precision. 5
Electronic health records (EHRs) are sources for a wide range of studies to plan and evaluate strategies for identifying and preventing disease. However, due to the exponential increase in health records, manually reviewing that amount of information is a very expensive and time-consuming task. Therefore, providing systems capable of automatically identifying and classifying entities or key phrases of interest in these records is a vital task.
One of the biggest challenges of this shared task is that there are a large number of entities or key phrases of interest, but they are often unbalanced in the text. All this, together with the presence of nested, discontinuous or overlapping entities, results in di culties in classifying them correctly.
In this document, we describe our system involved in the sub-task A proposed by eHealthkd-2019. It exploits the NeuroNer tool, a tool based on deep learning with bi-directional LSTM and CRF layers for the NER task. Considering the challenges described above, our system achieves a micro-F1 of 76.63% on the test equipment.
For future works, we plan to explore other deep learning architectures as
well as exploiting pre-trained word embedding models, as well as other types of
embeddings such as sense embeddings [
Funding: This work was supported by the Research Program of the Ministry of Economy and Competitiveness - Government of Spain (DeepEMR project TIN2017-87548-C2-1-R).