We present the work carried out by the HapLap group in the in the subtask B of the eHealth-KD 2020 competition. Relation extraction was addressed with a pipeline system that makes use of a Joint ABLSTM neuronal network together with a pre-process and a post-process phase. We obtained a result of 0.316 in Scenario 3. We present the work carried out by the HapLap group in the eHealth-KD 2020 task [1]. In this third edition the purpose of the task is to automatically extract knowledge, represented by means of thirteen semantic relations, from Spanish electronic health documents. We have taken part in the optional subtask B: the input is a plain text with entity annotations in a BRAT file and the output is the previous BRAT file with both the entities and relations. To address this, we have implemented a pipeline system that makes use of a Joint AB-LSTM neuronal network together with a pre-process and a post-process phase.
In the last years various competitions related to relation extraction have been emerging such as:
Semeval 2018 task 7 [
For this work we have divided the system into three phases: First the pre-process, where we adapt the data format to use with the Joint AB-LSTM. After that we have the training phase, where we train and evaluate the neural network and we get the prediction. And after getting the predictions we have the post-process, where we convert those predictions into the data format that is used in the competition.
In the preprocess we do the following operations: • Convert the input from the Brat standof format to the format used in the eHeathKD 2019 challenge. • Convert the data in the eHealthKD 2019 format into the format used by the Joint AB-LSTM. • Create the NO_RELATION relations.
In the first part of the system we have pre-processed the input relations. We have converted the Brat Standof input relation-format (also referred to as ann) to the format used in the previous eHealthKD 2019 competition by means of the ann2txt scripts ( https://github.com/ knowledge-learning/ehealthkd-2019/blob/master/scripts/ann2txt.py) provided there. Next, we needed to adapt it to what the Joint AB-LSTM requires. Three programs have been implemented for the pre-processing and their code has been posted on GitHub (https://github.com/Porobu/ HAPLAP-MAL). These three programs load the instances that are in the eHealthKD 2019 data format and they join them into a single file.
In an attempt to enable the neural network learn to discriminate between positive and negative relations (absence of relation), both types of instances should be provided in the inference stage. To this end, in the pre-processing an auxiliary relation class, NO_RELATION, was also created. A critical point, hence, is how to choose instances that contain pairs of entities that could be related and, thus, are candidate relations and label them as negative instances. Both the selection and the proportions might be crucial. We have used a simple way of choosing them, that only creates negative (NO_RELATION) relations between entity pairs that have at least one positive relation instance in the data set. To further reduce the negative relations, we have only created these between entity pairs in the same sentence.
At this stage we have a set of data with the candidates marked as either related or not-related. At this point a multi-class approach enables us to predict whether a candidate pair is related with some of the relation-classes available (including NO_RELATION). This was, indeed, our approach-1: a pair of entities that could be related (are a relation-candidate) are directly classified by means of the Joint AB-LSTM.
Needless to say, in the aforementioned sample negative instances substantially exceed the positive ones leading to skewed class distribution. In table 1 we can see the number of positive and negative relations in our training and development data sets. We have to remember that in our multi class classification approach (approach 1) the positive relation number contains all the thirteen classes, further skewing the data. Inference tends to be biased towards majority class. To cope with this we proposed to tackle the classification in two stages (our approach-2): • In the first phase we have created the binary data set, and all the positive relations (target, causes...) have been grouped in the RELATION class. In this phase we filter all the negative relations, to reduce the imbalance. • In the second phase we have now only the data set with the positive relations (arg, target, subject...), and we train the system to predict the relation.
Both approaches (and both phases in the second approach) were implemented by means of the Joint AB-LSTM approach. Further details are given in the following section.
After pre-processing the instances we load them into the Joint AB-LSTM neural network. The Joint AB-LSTM neural network has been inplemented by using Tensorflow. The network also does its own pre-processing. First all tokens are lower-cased.
The network employed word-embeddings as the main feature. For this work we have used pretrained embeddings from the clinical domain. The embeddings have been trained in corpora that consists of EHRs (electronic health records) that are not publicly available due to confidentiality issues. Other choices might have resulted more appropriate than ours since the amount and type of data employed has a big impact on the resulting embeddings. Apart from the wordembeddings, the network employs another powerful feature: the distance-embeddings. The distance is simply computed as the number of tokens between each annotated word in the sentence and the target word entity.
Having the relations completely pre-processed, the neural network is trained. This network combines two widely used neural networks in NLP: a Bi-LSTM with max pooling and an attentive Bi-LSTM. The Joint AB-LSTM is fed with the pre-processed sentences, their entities and relations between those, and the previously created distance embeddings.
We have optimised two hyper-parameters of the neural network, the dropout and learning rate to get the final model. We have trained the model with a mixture of the eHealthKD 2019 train+dev and the eHealthKD 2020 datasets, and we have used the eHealthKD 2020 dev dataset as validation. Note that this optimisation has been done over the so called multiclass dataset (approach 1), not over the binary dataset (approach 2). After doing the optimisation, we set 0.001 as the learning rate, and we used no dropout.
After getting the predictions from the neural network, we postprocess them to get the output relations in the Brat Standof format, respecting the IDs if the gold entities.
As described in section 3.1, we provided two diferent approaches. The results achieved with each of them are given in table 2.
Approach 1 outperforms Approach 2 in terms of precision but with the recall occurs the opposite. Nevertheless, for both approaches the F1-measure has the same value. Relation extraction was addressed with a neural approach, Joint AB-LSTM network. We applied two simple pre-processing approaches to get both positive and negative instances. This stage might result naive for the way in which the sampling was carried out and the proportions selected. We explored two pre-processing approaches: a straight one, approach 1, which just copes with multi-class problem; a filtered one (approach 2) that tried to get rid of negative candidates prior to the multi-class stage. None of them surpassed the other significantly. For future work, we should explore the embeddings provided to the network. Embeddings are the main source of knowledge in this stage with limited training sets and was proven significantly influential in related works.
This work was partially supported by the Spanish Ministry of Science and Technology PADMED (PID2019-106942RB-C31) and by the Basque Government (IXA IT-1343-19 and a Grant for the student Sergio Santana published in the 12/03/2020 BOPV).