This paper describes our approach presented for the eHealthKD 2019 challenge. Our participation was aimed at testing how far we could go using generic tools for Text-Processing but, at the same time, using common optimization techniques in the eld of Data Mining. The architecture proposed for both tasks of the challenge is a standard stacked 2-layer bi-LSTM. The main particularities of our approach are: (a) The use of a surrogate function of F1 as loss function to close the gap between the minimization function and the evaluation metric, and (b) The generation of an ensemble of models for generating predictions by majority vote. Our system ranked second with an F1 score of 62.18% in the main task by a narrow margin with the winner that scored 63.94%.
This article describes the model presented by the coin- ipper team for solving the
shared task presented in eHealth-KD 2019 challenge [
The tasks proposed are similar to previous competitions such as Semeval-2017
Task 10: ScienceIE [
As the organizers of the challenge state, although this challenge is oriented to the health domain, the kind of knowledge to be extracted is general-purpose. With this idea in mind, we propose a model that avoids the use of knowledge or tools speci c to the domain, easing the portability of the model to new domains and tasks. 2
The eHealth-KD 2019 challenge involves two subtasks: subtask A) Identi cation and classi cation of key phrases (four di erent types of units), and subtask B) Detection of semantic relations among them (thirteen semantic relations).
Each subtask has been evaluated individually and also as components of a pipeline. For this purpose, three evaluation scenarios have been proposed. Scenario 1 requires subtasks A and B to be executed sequentially in a pipeline. Scenario 2 only evaluates subtask A and Scenario 3 only evaluates subtask B.
A training set that contains a total of 600 sentences was provided to be used for the learning step. For the validation step, an additional set of 100 sentences was available.
A detailed explanation of the corpus data, subtasks and evaluation metrics
can be found in the overview description paper [
Our aim was to see how far we can get in tasks described above using generic NLP tools (non-speci c for the domain) and standard Data Mining strategies.
About the model to be learned, given the sequential nature of the data
(sentences), we proposed a stacked 2-layer bi-LSTM architecture. LSTMs have been
proved a successful approach in several Natural Language Processing tasks, such
as Named Entity Recognition (f.i. [
It is well known in Machine Learning that ensemble models have higher
performance than single models [
There are several ways to generate an ensemble of neural networks. Usually,
di erences in hyperparameters or di erent architectures are enough to ensure
diversity of models necessary for making ensemble methods pro table. In
our case we achieve diversity by using random initialization of models before
training [
There also exist a lot of di erent techniques to combine the output of di
erent models into a single prediction [
X yiy^i f ps = i2Y + X y^i = i2Y i2Y X(1 i2Y yi)y^i Notice that when predicted labels y^ are hard (0 or 1), these values are exactly true and false positive cases, respectively.
Replacing soft versions of true and false positives cases in F1 de nition (in the type I and type II error form), we have our surrogate F 1s function to maximize.
F 1s =
2 tps jY +j + tps + f ps =
2 Pi2Y + y^i jY +j + Pi2Y + y^i + Pi2Y y^i (1) Note that maximize F 1s implies increasing y^ of positive cases while decreasing y^ of negative cases in a proportional way to F1 function. Loss function to minimize will be 1 F 1s.
In the next sections we describe further details about how subtasks A and B are solved. 3.1
Subtask A consists in detecting key phrases in texts (from now on we will
abbreviate key phrase to KPhr) and their kind or class. For this task we set as input
only the sequence of words of each sentence and their Part-of-Speech (PoS).
Words are represented using FastText embeddings [
FastText embedding
POS1
… … … … …
PoS for sentences are generated using spaCy [
In the de nition of the output of the architecture we were inspired on image segmentation work in which, given an image, a mask over the image is generated that encodes for each pixel the class it belongs to. Similarly, for each sentence, we de ne as output a mask vector where for each word we have codi ed the kind of KPhr the word belongs to (or a label denoting that the word does not belong to any KPhr).
The architecture of the model to be learned consists of two stacked layers of bi-LSTM feed with the word embedding and the one-hot encoding of the PoS for each word of the sentence. We apply sequence Padding (with length parameter equal to the length of the larger sentence in data set) and Masking. Initially, output was de ned as a vector with the one-hot codi cation of the corresponding KPhr label for each word. However, we found empirically that learning a model to predict all kinds of KPhr was not very successful. So, we decided to split the task into learning models specialized in predictions for each kind of KPhr. With this modi cation, now the output of the architecture is a 0/1 vector with 1 denoting if a word belongs to the kind of KPhr the model is trained for, and 0 otherwise. Notice that with this modi cation, the use of F1 loss become even more important because the induced sparsity of labels.
The learning of each model is done by using surrogate 1 F 1s loss (as explained in previous section) with early stopping when F 1 score does not improve 0/1
0/1
POS1 Kind KPhr Source KPh
POS2 Kind KPhr Source KPh in 100 epochs on the development dataset, and returning the model that reaches the best F 1 score on development dataset. Default learning rate 0:001 with adam optimization is used. Batch size is 32 sentences. Neither dropout nor data augmentation is used. First bi-LSTM layer has 150 units in each direction. Second bi-LSTM layer has 32 units in each direction. The same architecture is used for all KPhr detection models without ne-tuning of parameters. See a graphical representation of the model in Figure 1.
As stated in the previous section, we create an ensemble of 15 models for each target KPhr that is used later to generate the nal prediction by majority voting. Diversity of models in the ensemble is obtained only from di erent initialization of weights. To avoid the vote of specially bad models, each model is applied to the development dataset and those models with an F1 score far below the average of the ensemble are removed from it.
Our approach has two obvious problems. The rst one is that, because the way we de ne segmentation, we don't know when a KPhr starts nor ends. A naive solution would be to join into the same KPhr consecutive words that are tagged with the same label. However this heuristic solution does not always work. We solve this problem by inspecting the PoS of consecutive words tagged with the same label. For each combination of PoS found, we compute how many cases covered are in fact cases where words belong to the same KPhr. From the statistics of these cases we create a set of rules that decide if two consecutive words with the same label really belong to the same KPhr or not.
The second problem is that having di erent models, one for each kind of KPhr, words could be labeled as belonging to di erent kinds of KPhr at the same time. So, in case of con ict among di erent ensembles, in order to decide the nal kind of KPhr, we count how many positive votes each ensemble has emitted and multiply it by a weighting parameter. The ensemble with the highest product is the winner and the word is labeled with the kind of KPhr the ensemble represents. Weights for each ensemble are found by applying the ensembles on the development dataset and grid-searching for the combination of weights that returns highest global F1. 3.2
Subtask B consists in detecting semantic relations (thirteen types of relations) between KPhr pairs identi ed in the previous task. For this task, given a relation rel and a KPhr considered as source for the relation, we try to identify another KPhr in the sentence with the role target for this relation rel. In order to learn models able to solve this task, we de ne an architecture that, given a relation rel and a source KPhr, has the following inputs: 1. Sequence of words in the sentence where source KPhr is found. 2. Part of Speech tags for each word in the sentence. 3. Mask denoting for each word the kind of KPhr it belongs to (codifying in 1-4 the kind of KPhr and 0 if it does not belong to any KPhr).
0/1 0/1 0/1
POS1 4. 0/1 Mask vector for each word with 1 denoting if the word belongs to the source KPhr, and 0 otherwise.
Words are represented as FastText vector embeddings and all other inputs are encoded using one-hot encoding. All vectors codifying the information of one word are concatenated. See Figure 2 for a representation of the architecture.
For each relation we build a Neural Network that consists of the embedding layers explained above plus 2 stacked bi-LSTM layers. As in subtask A, we use padding and masking. The output consists of a mask vector 0/1 with 1 only for those words who belong to a target KPhr of the relation rel of current KPhr as source. When no target KPhr exists for the current relation, output is all zeros.
When the model identi es at least one word of a target KPhr, we consider the whole KPhr detected. The output representation selected allows that in the same sentence one KPhr as source has several KPhr as target for the same relation.
All training parameters and procedures are the same as those used in subtask A. Again, the learning of each model is done by using surrogate 1 F 1s loss with early stopping when F 1 score does not improve in 100 epochs on the development dataset, and returning the model that reaches the best F 1 score on development dataset. Default learning rate 0:001 with adam optimization is used. Batch size is 32. Neither dropout nor data augmentation is used. First bi-LSTM layer has 150 units in each direction. Second bi-LSTM layer has 32 units in each direction. The same architecture is used for all relations without ne-tuning of parameters. For each relation, an ensemble of 15 models is built from which majority voting is implemented.
In some cases, ensembles of di erent relations identify the same pair of KPhr as positive cases. When this happens, we assign to the pair of KPhr the relation of the ensemble with highest precision on the development dataset. 4
In this paper we have presented a simple approach to solve the tasks proposed in this competition. The aim was to measure how far we can get with standard tools and simple algorithms but using common optimization techniques in the Data Mining eld. In particular, the architecture proposed for both tasks was a two level bi-LSTM to modelize each of the classes we wanted to capture. We didn't use state of the art language models like BERT, ELMO or GPT, neither specialized NER techniques, knowledge bases or taggers speci c for the domain (so results probably will be similar in other domains).
The main particularities of our approach are: (a) The use of a surrogate loss function of F1 metric to close the gap between the minimization function and the evaluation metric and (b) the generation of an ensemble of models for generating predictions by majority vote.
From results, we conclude that it is worth using the evaluation metric (or a
surrogate function of it) as loss function and also building ensemble models for
generating predictions. We did not do an exhaustive study of the improvement
in performance due to the use of a surrogate function of F1 as loss, but in initial
experiments we found a clear advantage of the surrogate function usage instead
of standard loss functions. We neither did an exhaustive study of the impact
in performance of learning ensemble models instead of learning single models,
however, we noticed that usually F1 scores obtained by the majority vote of
the ensemble was about 1-2 points higher than the best model of the ensemble
and 3-4 points higher than the average of the models of the ensemble. Note that
combination of outputs in the ensemble is made through simple majority voting
but more complex ensemble techniques could improve the performance of the
system even more. For a review of ensemble methods see [
The results show that our model achieves good precision scores for the three
scenarios but sometimes at the expense of lower recall levels than other systems,
specially notorious in scenario 3. The di culties our model had in identifying
some semantic relations may be due to the low number of examples of these
relations in the training dataset. To deal with this problem, we are planning to
use data augmentation techniques in order to increase the number of instances
of the less frequent relations. One of the techniques we want to explore is
backtranslation, used in text classi cation [
Funding: Neus Catala's work has been partially funded by the Spanish Government and by the European Union through GRAPH-MED project (TIN201677820-C3-3-R) and AEI/FEDER, UE.
Mario Martin's work has been partially supported by the Joint Study Agreement no. W156463 under the IBM/BSC Deep Learning Center agreement.