This article describes the proposed system by the VSP team in the Relation Classi cation subtask of the eHealth-KD Challenge 2019. The architecture is a bidirectional Recurrent Neural Network with LSTM cells (BiLSTM) that uses the word embedding, position embedding and entity type embedding of each word as inputs. Later, a Softmax layer classi es the relationships between entities in the sentences. The presented BiLSTM model reached a 49.33% in F1 for the Scenario 3 of the challenge (Relation Classi cation). Moreover, the system ranked third out of eight teams that participated in this subtask. The main advantage of this approach is that any hand-crafted feature is used because the system can extract the relevant features automatically.
The number of scienti c publications increased until 6% each year, whose largest
subject area is the biomedical domain [
The TASS-2018 Task 3 [
Nowadays, Recurrent Neural Networks (RNN) has shown good performance
for tasks where the data are sequential. Concretely, Long-Short Term Memory
(LSTM) cells are designed to capture the long distance dependencies between
words in the sentences. In Relation Classi cation, the bidirectional Recurrent
Neural Network with LSTM cells (BiLSTM) of [
This work describes the participation of the VSP in the Scenario 3 of the eHealth-KD Challenge 2019 that involves the classi cation between entities. The proposed architecture is a BiLSTM that generates the representation of the sentences with a relationship and a Softmax layer that classi es the categories of the relationships in one model. The system uses the representation of the words in the sentences together with the information of the entity types labels and the position with respect to the two interacting entities as embeddings. 2
An annotated dataset of Spanish eHealth documents was developed for the development and evaluation of the systems in the eHealth-KD Challenge 2019. This corpus was manually extracted from MedlinePlus documents using only the health and medicine topic written in the Spanish language. The data was divided into three datasets: training, development and test sets. The training set and development sets have 600 and 100 manually annotated sentences used for the learning step and the validation of the models, respectively. Additionally, the test set contains 100 sentences together with the annotations of the mentions for the evaluation of the Scenario 3.
The annotated relationship between entities are divide in four categories: { General relations: indicates general relationships between the entities, they are is-a, same-as, has-property, part-of, causes and entails classes. { Contextual relations: indicates the re nement of an entity, they are in-time, in-place and in-context classes. { Action roles: de nes the role plays related to an Action entity, they are the subject and target classes. { Predicate roles: de nes the role plays related to an Predicate entity, they are the domain and arg classes.
A more detailed description of the annotation guidelines can be found in [
Pre-processing phase The relationships in the eHealth-KD Challenge 2019 are asymmetrical, that is the two entities are related in one direction. For this reason, a pair of entities are annotated with two labels for both directions. Thus, a sentence with n entities will have (n 1) n instances. Each instance is labelled with one of the thirteen classes de ned by the task. In addition, a None class is also considered for the non-relationship between the entities. Table 1 shows the resulting number of instances for each class on the train, validation and test sets.
Once the instances are extracted from the documents, the sentences are
tokenized and cleaned similarly to [
Relationship between entities Instances after entity blinding Label (asma ! enfermedad) 'el entity1 es una entity2 que entity0 las entity0 .' is-a (asma enfermedad) 'el entity2 es una entity1 que entity0 las entity0 .' None (asma ! afecta) 'el entity1 es una entity0 que entity2 las entity0 .' None (asma afecta) 'el entity2 es una entity0 que entity1 las entity0 .' subject (asma ! v as respiratorias) 'el entity1 es una entity0 que entity0 las entity2 .' None (asma v as respiratorias) 'el entity2 es una entity0 que entity0 las entity1 .' None (enfermedad ! afecta) 'el entity0 es una entity1 que entity2 las entity0 .' None (enfermedad afecta) 'el entity0 es una entity2 que entity1 las entity0 .' None (enfermedad ! v as respiratorias) 'el entity0 es una entity1 que entity0 las entity2 .' None (enfermedad v as respiratorias) 'el entity0 es una entity2 que entity0 las entity1 .' None (afecta ! v as respiratorias) 'el entity0 es una entity0 que entity1 las entity2 .' target (afecta v as respiratorias) 'el entity0 es una entity0 que entity2 las entity1 .' None
In the corpus, there are some instances with multiple annotations, the vast of them are annotated like target and subject. Only one of these classes is kept because the proposed system does not tackle the multi-class classi cation. Additionally, there are entities with discontinuous tokens that have some overlapping parts with other entities. Thus, only the non-overlapping parts of the mentions are kept given that the entity blinding process cannot deal with this kind of entities. 3
This section presents the BiLSTM architecture which is used for the Scenario 3 of the eHealth-KD Challenge 2019. Figure 2 shows the RNN model where the inputs are the preprocessed sentences and it generates a prediction label of the relationship between the marked entities. 3.1
Word table layer Firstly, the preprocessed sentences are transformed into an input matrix for the RNN architecture. Padding is added to all the sentences until reaching the maximum length of a sentence in the dataset (denoted by n). Thus, the sentences shorter than n are padded with an auxiliary token "0 ".
Each word in the sentences is represented by its word and entity type embeddings. These embeddings are extracted from the word embedding matrix We 2 RjV j me where V is the vocabulary size and me is the word embedding
El <e1>asma</e1> es una <e2>enfermedad</e2> que <e0>afecta</e0> las <e0>vías respiratorias</e0>. n |V| el entity1 es una entity2
que entity0
las entity0 0 … 0
Preprocessing Sentence: el entity1 es una entity2 que entity0 las entity0 Enddtiissitttyaanntyccpee21e::: -O-14 Co-n03cept-O12 -O21 Con03cept O41 Ac25tion 6O3 Con47cept
We
WEntity
Wd1
Wd2 me Word embeddings
md Entity type Position embeddings embeddings
md Position embeddings |E| mEntity
X Look-up table layer 2n-1 …
S Recurrent layer z m
k Ws
o Pooling layer
Softmax layer with dropout dimension and the entity embedding matrix WEntity 2 RjEj mEntity where E is the vocabulary size of the entity types and mEntity is the entity type embedding dimension, respectively.
In addition, two position embeddings are concatenated to these vector in order to represent the relative position of each word with respect to the two interacting mentions. These distances are mapped into a real value vectors with the two position embedding matrices, Wd1 2 R(2n 1) md and Wd2 2 R(2n 1) md where md is the position embedding dimension.
Finally, each word in the sentence is described by a vector giving a matrix
that represents the sentence of the relationship X 2 Rn (me+mEntity+2md).
The resulting matrix is the input for the Recurrent layer. In this system, LSTM
cells [
it = (Wi [ht 1; xt] + bi) c0t = tanh(Wc [ht 1; xt] + bc) ct = ft ct 1 + it c0
t ot = (Wo [ht 1; xt] + bo)
ht = ot tanh(ct) where Wf , Wi, Wc and Wo are the weights and bf , bi, bc and bo are the bias term of each gate.
Later, the current output ht is represented with the hyperbolic function of the cell state and controlled by the output gate. Furthermore, another Recurrent layer can be applied in the other direction from the end of the sequence to the starting word. Computing the two representations is bene cial for extracting the relevant features of each word because they have dependencies in both directions. Finally, the output vector of both directions is concatenated giving an output matrix S 2 Rn m where m is the number of output dimensions for the Recurrent layer. 3.3
Pooling layer The pooling layer extracts the most relevant features of the output matrix in the Recurrent layer using an aggregating function. In this model, the max function is selected to produce a single value for each output as zf = maxfsg = maxfs1; s2; :::; sng. Thus, the vector z = [z1; z2; :::; zm] is created, whose dimension is the total number of lters m representing the relation instance. 3.4
Softmax layer Before the classi cation, a dropout is applied to prevent over tting. Thus, a reduced vector zd is obtained from randomly setting the elements of z to zero with a probability p following a Bernoulli distribution. After that, this vector is fed into a fully connected Softmax layer with weights Ws 2 Rm k to compute the output prediction values for the classi cation as o = zdWs + d where d is a bias term; in this case, there are k = 13 classes in the dataset and the "None " class. At test time, the vector z of a new instance is directly classi ed by the Softmax layer without a dropout. 3.5
Learning The following BiLSTM parameters !b f , W!i, !b i, W!c, !b c, W!o, !b o, Wf , b f , Wi, b i, Wc, b c, Wo, b o) need to = (We, WEntity, Wd1, Wd2, Ws, d, W!f , be learned in training the network. To this end, the conditional probability of a relation r obtained by the Softmax operation as is minimized by the negative log-likelihood function for all instances (xi,yi) in the training set T as follows p(rjx; ) =
exp(or)
Plk=1 exp(ol) J ( ) =
T
X log p(yijxi; )
i=1
In addition, the stochastic gradient descent over shu ed mini-batches and the
Adam update rule [
The weights of the BiLSTM model were learned on the training set during 25
epochs using mini-batches and Adam update rule [
C C + S
R =
C C + M
F 1 = 2
P R
P + R where Correct (C) are the relations that matched to the test set and the prediction, Missing (M) are the relations that are in the test set but not in the prediction, and Spurious (S) are the relations that are in the prediction but not in the test set.
Table 4 presents the results of the BiLSTM for all the classes and the nal results in the Scenario 3 (Relation Classi cation). It can be observe that the number of Missing is higher compared to the number of Spurious which makes a low Recall in almost all the classes. In general the Action and Predicate roles are classi ed better than the General or the Contextual relations. In addition, the classes with less than 100 examples in the training set, such as same-as, has-property, part-of, entails or in-time, has a low performance because the architecture can not extract the relevant features of these classes compared with the more than 20,000 examples of the None class. Thus, the vast of them are classi ed as None and taken as Missing.
Eight teams participated in this subtask being the BiLSTM model the third highest F1. The performance of the proposed system is very promising obtaining 49.33% in F1 as o cial results in Scenario 3 of the eHealth-KD Challenge 2019 that has thirteen relation categories. One of the main advantage of this approach is that it does not require any external knowledge resource.
This paper presents a BiLSTM model for the eHealth-KD Challenge 2019 Scenario 3 (Relation Classi cation of Spanish eHealth documentss). The o cial results for the proposed system are very promising because the model is a simple architecture that does not need expert domain knowledge or external features. However, the performance of the method is very low in Recall measure because there are a high number of Missing instances that are classi ed as the None class. One possible solution could be the creation of four independent classi cation systems for the four kind of classes taking into consideration the entity types of the mentions. Thus, each system could be more balanced and specialized in each type of relationships. Moreover, it is hard for the system detecting the directionality of the relationships. For this reason, the creation of a reverse class for each relation type could help to the architecture in order to distinguish the direction of the relation.
As future work, the aggregation of external feature as embedding vector, such as Part-of-Speech tags, semantic tags or syntactic parse trees, could improves the representation of each word and increase the performance. Furthermore, the exploration of deeper layers in the Recurrent layer is proposed to be included in the BiLSTM model for the classi cation of relationships in Spanish eHealth documents.