This paper describes the solution presented by the UH-MAJA-KD team at IberLEF 2020: eHealth Knowledge Discovery Challenge. Separate strategies were developed to solve Tasks A and B, both based on Deep Learning models using contextual embeddings obtained from a pretrained BERT model, and some other syntactic features. We propose a strategy using a hybrid model for Task A that uses Stacked Bidirectional LSTM layers as contextual encoders, and linear chain Conditional Random Fields as tag decoders. The system addresses Task B in a pairwise-query fashion, encoding information about the sentence and the given pair of entities using syntactic structures derived from the dependency parse tree, by the means of LSTM-based Recurrent Neural Networks. The output is obtained scoring every possible relation via a Multilayer Perceptron with a sigmoid activation function. Our model was able to get a high performance in all four tasks of the competition. The system was ranked third in the main evaluation scenario, with a 0.001 diference with the second place. Additionally, it was ranked second in the evaluation responsible for measuring the performance in Task B, considered the hardest one in previous editions of the challenge.
ory (BiLSTM) layers as contextual encoders, because of the sequential structure of the input
and its widely use in the literature [
The rest of the paper is organized as follows. Section 2 explains in detail the proposed system. The results of the model in the several scenarios evaluated during the eHealth-KD 2020 event are presented in Section 3. Section 4 analyses briefly matters of interest related to the development and performance of the models. Finally, the conclusions of the work are shown in Section 5.
The system proposed solves both tasks separately and sequentially. Thus, independent models were defined to solve NER and RE problems.
The NER task is posed as a tag prediction problem that takes the raw text of the input
sentence and outputs two independent tag sequences: one in the BMEWO-V tag system for
entity prediction [
Given the target sentence and the highlighted entities input as raw text, some preprocessing is done in order to derive useful structures from such text. Since both models make use of word-piece information, the input sentence must be tokenized first. Other preprocessing steps include character-level word decomposition, syntactic features extraction and dependency parsing.
To obtain a representation of the corresponding inputs, the models make use of the following features for each word: Contextual embedding: BERT-based contextual embeddings with no further hypertuning.
Due to BERT model’s tokenization algorithm, a certain strategy is needed to merge words divided into multiple BERT tokens (e.g, word cáncer might be divided in [cán, cer]). In our case, it is done using the mean of the given vectors. Each model uses the concatenation of a number of BERT output vectors. Character embeddings: CNN-based character embeddings. The input to such CNN is a sequence of alphabet indexes, those of the characters contained in the word. POS-tag and Dependency embeddings: Embeddings intended to encode word-level syntactic features such as the POS-tag of the given word and the dependency with its ancestor in the dependency parse tree.
BMEWO-V and Entity Type tags: BMEWO-V and entity type tags are used in RE task and are obtained from Task A model outputs.
Contextual embeddings are pretrained with no further hypertuning; whereas the remaining embeddings’ weights are optimized when training the corresponding model.
The model receives the sentence as a sequence of words vectors . A distributed representation of each word is obtained concatenating contextual, character and POS-tag embeddings, as described in the previous subsection.
At a second level, the sequence of tokens is processed in both directions by a BiLSTM layer, resulting in two sequence vectors. The vectors on complementary positions of the two sequences are concatenated resulting in a new sequence with contextual-dependent vectors assigned to each token in the sentence. This sequence is looking to encapsulate semantic dependencies between the tokens of the sentence. The output sequence of the first BiLSTM is processed in both directions by a stacked BiLSTM on top of the first one, getting more representational power and resulting in the sequence of vectors ′.
′ = = ( ) ( )
Sequence ′ is fed into two linear chain CRF layers, that output the most likely tag sequences
according to the Viterbi algorithm [
( ′) = = Figure 1 shows the described architecture.
The first CRF layer produces a sequence of tags in the BMEWO-V tag system. This classification corresponds to B for the begin of an entity, M for tokens in the middle, E for the ending token, W in the case of tokens that are an entity themselves and O for tokens that do not represent anything. It also takes into account the possibility of overlapping entities, including the tag V in such cases.
A process is necessary to transform the tag sequences got from the CRF layers into a list
of entities expected as output for Task A [
In accordance to Spanish correct use, the set of tag sequences that must be interpreted as a group of discontinuous entities were reduced to those that match the regular expressions (V+)((M*EO*)+)(M*E) and ((BO)+)(B)(V+). The former corresponds to entities that share their initial tokens, and the latter to those that share their final tokens. These two capture most of the desired discontinuous entities. Among the examples of the former case, it is found the fragment cáncer de pulmón y de mama, tagged as V-M-E-O-M-E, where entities cáncer de pulmón and cáncer de mama are found. And, as example of the latter, the fragment tejidos y órganos humanos, tagged as B-O-B-V, where entities tejidos humanos and órganos humanos are found. When a match is detected and the entities are extracted, all the tags in that fragment are set to tag O.
After the detection of possible discontinuous entities, the second stage starts assuming all the remaining entities appear as continuous sequences of tokens. To extract continuous entities, an iterative process is carried on over the tag sequence produced by the model. Due to limitations in the BMEWO-V system, the procedure also assumes that the maximum overlapping depth is 2. Assuming otherwise only makes the process more ambiguous and does not capture much more information since deeper overlappings are not frequent on the training and development collections. Given this, along with the procedure, two in-construction entities are maintained. In each iteration these two entities are created, extended or emitted in accordance to rules defined considering only the previous and the current tag.
The most complete information for solving the RE task is found in the whole input sentence.
However, some authors claim that the dependency tree associated with the input sentence
condenses the most important information, and discards the misleading [
According to observations, highlighted entities in the sentence collections are complete
nominal phrases (or at least sub constituents of them). Some of the criteria taken into consideration
to establish a dependency relation with a header H in a syntactic construction C, is the fact
that H could replace C [
Another important definition, vastly used in literature to address this task, is the dependency path between two tokens 1, 2. From now on it is going to be referred to as ( 1, 2).
The before-mentioned structures are fed into a Deep Neural Network that outputs a vector whose length is the same as the relations set. Each component of such vector is a score that measures the strength with which the corresponding relation is present between the input entities.
To do so, the model first encodes each of the structures 1 , 2 and ( 1 , 2 ) in a vector. Either 1 and 2 or ( 1 , 2 ) are formed by words from the input sentence. A distributed representation of each word is obtained concatenating contextual, character, POS-tag, dependency, BMEWO-V and entity type embeddings, as described on the previous subsection.
To compute the output vector, a BiLSTM layer encodes the sequence of vectors associated to the words in ( 1 , 2 ) to include bidirectional information in the representation.
= ( ( 1 , 2 ))
This output is fed into a unidirectional LSTM layer so as to emphasize the direction of the potential relation, processing the sequence from the origin to the destination. This results in a vector encoding the information present in ( 1 , 2 ).
= ( )
At the same time, a ChildSum Tree-LSTM [
1 = ( 1 ) 2 = ( 2 )
Vectors encoding the input structures are concatenated. The final output is obtained by applying a sigmoid function to a linear transformation of it.
1For simplicity, any notation related to the sentence is gonna be omitted as long as is clear by the context. prematura with the highlighted entities cáncer de pulmón and muerte.
= [ 1 ; 2 ; ] = ( ( ) + ( ))
According to the scores present in the output vector , if any of its components exceeds a given threshold, then the relation with the maximum score is predicted. If none of the scores is greater than such threshold, then no relation is reported. The threshold value is added as an hyperparameter and optimized using the development collection. Notice that this approach allows us to disregard the use of a fake relation none.
For both subtasks, the training procedure was carried out using only the training collection provided to contestants.
Since the CRF layer is intended to maximize the probability of obtaining a desired tag sequence Then, that probability is defined as the normalized exponential: given an input feature vector , the Task A model is trained to minimize the negative log of the probability ( | ). Let
and be the CRF emissions and transition matrixes respectively. ( | ) =
exp(∑ =1 ( , ) + ∑ =1 ( , +1))
−1 ( ) being a normalization factor depending on the input vector . And the loss function is defined in terms of and as follows:
( , ) = − ( ( | ))
In the case of Task B model, since each output component is independent to each other, the model is trained to minimize a binary cross-entropy function over the output vector. Let be the number of relations, the output vector and the target vector, the loss function is computed as follows:
1 (, ) = ∑ [ ⋅ log + (1 − ) ⋅ log(1 − )]
1≤ ≤
As explained before, the model output does not make use of the fake none relation. A negative sampling strategy is used so that the model is optimized with examples where no relation is present. A negative sample is nothing more than a training example where the target output is the null vector. Such sampling is performed using a fixed proportion of unrelated entity pairs.
Dropout strategies were used during the training procedure in both models to reduce overfitting. For Task A, two dropouts layers were stacked after the first and the second BiLSTM, and a spatial dropout 2D was added after the CNN layer used to compute the character embedding of words. In Task B model, three dropout layers were stacked after BiLSTM, LSTM and TreeLSTM layers respectively.
The number of epochs was selected empirically, based on the convergence of the models, as learning curves showed. For hyperparameter tuning and model selection, a cross-validation process was carried out using the development collection. Table 1 shows the hyperparameter setup for both models.
The systems were implemented using Python programming language, with PyTorch (v1.4.0) library as the deep neural networks framework. BERT-based contextual embeddings were obtained from the bert-multilingual-uncased pretrained model, using the Python library pytorch-pretrained-bert2 (v0.6.2). POS-tag and dependency tree were obtained using the Python library spaCy3 (v2.2.1), specifically the model es_core_news_md (93MB).
Both of the models were trained in a personal computer with the following features: Intel(R) Core(TM) i7-6500 CPU at a frequency of 2.50GHz, with an installed memory of 8.00 GB with no GPU available for CUDA. The total traning time for the entity model took less than 5 hours, whereas the relation model was close to 12 hours.
The evaluation in both tasks was carried out using the annotated corpus proposed in the
challenge. The results were measured with a standard F1 measure as described in detail in the
2https://pypi.org/project/pytorch-pretrained-bert/
3https://spacy.io/
challenge overview [
Table 2 presents the oficial results of the competition, given by the evaluation scenario 1. As shown, our system was ranked as third best, achieving an overall F1 score of 0.625, quite close to the second best, and rather far from the first and fourth ranked systems.
Table 3 shows the results corresponding scenarios 2 and 3, where Task A and B were evaluated independently. With F1 scores of 0.814 and 0.598, our system was able to reach the fourth and second positions on Task A and B evaluation scenarios, respectively.
Finally, an additional transfer-learning scenario was proposed in this edition of the challenge. Scenario 4 evaluates the generalization capabilities of the systems to general-topic domains. Table 4 compares the results of the participant systems in this evaluation scenario.
As can be seen, all systems performed worse in this scenario that in scenario 1, but the ordering remained almost identical. Ours was ranked third as in scenario 1, with a F1 score of 0.547. Team Vicomtech Talp-UPC UH-MAJA-KD IXA-NER-RE UH-MatCom SINAI HAPLAP baseline ExSim
F1 The BERT-based contextual embeddings proved to contain useful features for solving both NER and RE tasks. We also experimented using pretrained word embeddings trained on a medicalcontent corpus extracted from Wikipedia, and the model with the BERT features outperformed
Precision
Recall
Team
Precision
Recall the latter in both tasks. The combined usage of them proved to be rather inefective. Also, cased and uncased BERT models were tested and the cased model showed better results in the NER model. However, the RE task experienced a rather insignificant decay in performance. This results are based on the model selection process carried out using the development collection.
It is worth mentioning that our experiments showed that BERT is not enough to solve neither of the tasks as described in eHealth-KD 2020 challenge. BERT-only based models (i.e., models with BERT-based inputs and a corresponding CRF or MLP output layers), failed to perform well in both tasks.
For both cases, all the features used as inputs, as described in Section 2, proved to be determinant to achieve top performance. In the particular case of the RE model, aside from BERT-based contextual embeddings, information obtained from the NER task about the entities (i.e., the BMEOW-V tag and the entity type), are the most influential.
Finally, regarding the training process, it is worth noting the fact that the training time of the NER model is significantly shorter than the corresponding to the RE model. This is something expected, since our RE approach defines as a train example a sentence and a pair of entities (thus, much more training examples). Also, as for previous eHealth events, systems still perform significantly poorer in the RE task. These facts might lead to the conclusion that many findings are yet to be done.
In this work were described the system proposed by UH-MAJA-KD team at the IberLEF eHealthKD 2020: eHealth Knowledge Discovery challenge. For Subtask A was proposed a hybrid Stacked-BiLSTM-CRF model, using BERT pretrained contextual embeddings and other syntactic features. This model obtained competitive performance in Scenario 2, where it was located in fourth place with a small diference with respect to the top 3. Task B was addressed in a pairwise-query fashion, encoding information about the sentence and the given pair of entities using syntactic structures derived from the dependency parse tree, and by the means of LSTMbased RNN. The output is obtained scoring every possible relation via a Multilayer Perceptron with a sigmoid activation function. Our model obtained the second place in Scenario 3. The system reached the third position in the overall standing (Scenario 1), and also in the transfer learning scenario (Scenario 4).
It is proposed as future work to fine-tune BERT embeddings along with the training process of the proposed models, looking for contextual embeddings to be trained specifically for these tasks. Also, to use domain specific features like gazetteers looking to improve performance in the health domain. Finally, to develop and evaluate the usage of joint models for solving both tasks.