.This article presents the design choices and training strategy behind the model presented by the JAD team for at eHealth-KD Challenge 2021. The model consist of identifying key-phrases and relations among them using a predefined system. It was a simple model that summarizes some parts of a general approach to NLP problem. The system is easy to train and test using cloud services like Google Colab. It did not perform very well at the competition. The paper includes possible improvements.
This article describes the design choices and training strategy behind the model
presented by the JAD team for eHealth-KD Challenge 2021[
Our team presented a system with a fairly simple architecture: a pre-trained
multilingual BERT[
The rest of the paper is organized as follows. Section 2 explains in detail the proposed system. The results of the model in the several scenarios evaluated during the eHealth-KD 2021 event are presented in Section 3. Section 4 analyses briefly matters of interest related to the development and performance of the models. Finally, the conclusions of the work are shown in Section 5. 2
The output consists of a simple array/tensor of binary values with predefined length. This array may be partitioned in two sub-arrays, the first consisting in the key-phrases annotation and the second consisting in the relations annotations. • i: Represents the key-phrase label position in a custom fixed-length array of custom labels • j: Represents the word position in the sentence with a maximum position of the 100th word If an element is true, it is possible to find the word and the label that matches it. • i: Represents the relation label position in a fixed-length array of custom labels • j: Represents the word position in the sentence with a maximum position of the 100th origin word • k: Represents the word position in the sentence with a maximum position of the 100th destination word If an element is true, it is possible to find the origin word, destiny word, and the label that matches it. Output processing. This output format must be parsed to the competition output format and vice versa. There are certain problems that were addressed: • The model output just considers individual words as key-phrase, and the competition output may require several words: This was addressed including a new type of relation called samebox which is reflexive and links every word that should be in the same key-phrase. Also several words key-phrase’s relations were copied to every single word key-phrase. • Inconsistencies in the model output: This was addressed by a permissive parsing to the competition output that solved many inconsistencies by itself. 2.2
Training Our system was trained using a Google Colab notebook with GPU in Python with Keras and Tensor Flow with a binary cross entropy loss function. The training task with all its hyperparameters is available in Google Colab for reference at https://colab.research.google.com/drive/1L0AG1fD9dHzVlv8icOc1OruCiedggKG?usp=sharing
Figure 4 shows the accuracy achieved by our model after each training epoch. Accuracy can not be observed well enough but due to the elevated amount of false values in the target output is reasonable that after not many epochs is difficult to interpret improvement due to small improvements in the terms of less than 1%. Figure 5 shows the loss achieved by our model after each training epoch. Loss is a little easier to observe in these graphics. It diminishes with time, but dramatically faster in the training set, and slower in the evaluation set. Both sets get closer to 0 with many epochs, but after a while, around 130-145 epochs there is no improvement in evaluation set. 3
The evaluation in both tasks was carried out using the annotated corpus proposed in
the challenge. The results were measured with a standard F1 measure as described in
detail in the challenge overview [
From tables 1 to 3 our team always ranked 8th according to F1. Results in Task A were superior to results in Task B. None of our team results performed better than baseline. 4
The system achieved poor results in the challenge. The system was non-performant finding relations, and a little better labelling key-phrases. In both cases it was worse than baseline. These results could be due to several reasons:
One of the reasons could be simplicity of the model. If we add recurrent or convolutional layers the results may improve. Such improvement in the long run in the evaluation set could also indicate that the evaluation set was too similar to the training set. The way that the outputs were modeled might be improved to include less negative values. If we divide the model in two separate models also the results could improve.
The accuracy and loss achieved in training was not too good but in the final dataset the model performed poorly. In fact these metrics were a lot better in the training set than in the evaluation set, but the most relevant is the evaluation set because it contains data that our model has not trained on. This last one metrics may look well but in reality, these are not that good. Binary accuracy got to 0.997, but we should remember our output format is large in parameters size so this could imply not such a great performance.
This paper describes the system presented by team JAD, in the eHealth-KD Challenge 2021. A deep-learning model was trained and ensembled to automatically extract relevant entities and relations, from plain text documents. The results achieved by the system in the challenge were not outstanding, ranking last in the main task, being better at classifying entities, but still worse than baseline.
The main goal of our team was not winning the competition but to build knowledge and a general and simple model easy to understand, implement, train and run. This goal was completed mostly. The power of a simple model was overvalued. It would be interesting to add some recurrent layers, re-train BERT in a more specific dataset, and reducing output size or changing output format. 6
We thank all the organizers, staff and reviewers of the eHealth-KD Challenge 2021. Special thanks for the professors of the AI Department of MATCOM-UH.