The IberLEF eHealth-KD Challenge 2021, held at IberLEF 2021, proposes two subtasks to encourage the development of systems for automatically extracting knowledge from unstructured Spanish eHealth texts. I only participate in subtask A: Entity Recognition. This subtask aims to identify all the entities and their types for the given eHealth documents. This paper describes the system presented by team-Maoqin in the challenge. Several deep learning models are used in plain text documents, such as BERT-CRF, BiLSTM-CRF. Only the best result running in the local has been submitted. But the nal result of my method is indeed very low, with a score of 0.173 (F1), ranking 9th on the leaderboard. Additional work needs to be done to improve the nal result and complete the subtask B: relation extraction.
In recent years, the amount of medical documents produced by the scienti c
community has been increased. Those texts combine many corporates, it is
necessary to extract useful knowledge with automatic methods. Therefore, various
competitions have been held in the past years such as task 7 of SemEval 2018 [
The pre-trained and deep learning models have shown excellent performance
in many NLP tasks such as text classi cation, reading comprehension and
question answering [
The rest of this article is arranged as follows. Section 2 introduces the di erent architectures used by the system. In Section 3, the formal results obtained in the challenge are introduced. In Section 4, some ideas on improving the quality of each strategy and some un nished experiments will be shared. Finally, Section 5 introduces the conclusions of this paper and some opinions for future work. 2
This part is used to comprehensively describe the model mentioned and how to
process input and output data. Input handling, where I adjust the data format
for use in the model. The description of the structure includes architecture and
parameters set. After getting the prediction result, the result format should
comply with o cial rules.
2.1 Input Handling
The o cial sentence example is shown in gure 1 and the corresponding format
should be predicted as gure 2. The training and development corpora are both
provided in Brat format which consists of an id increasing by row, the entity
type, the next two numbers indicate the span of this entity and the word in
last. If adjacent words belong to the same entity, write them on one line, the
span should be separated by semicolons. The neural network only accepts token
level so the documents will be converted to output the models can use with the
provided txt and ann le. The input le is nally changed into BIO format and
input into the model. Among them, B means beginning, I means inside, and O
means non-entity. This scheme is the most popular in the NER task although it
presents problems when the entity contains discontinuous tokens [
BERT-CRF Model The BERT model has performed excellently in many areas in NLP. The tokens are fed into the BERT model to obtain their contextual embeddings. These embeddings are passed to a classi cation layer that emits logits with the prediction about each token being or not an entity of a certain type. Each entity type has B and I (8 in total) plus O, P AD, SEP , and CLS, so the input contains a total of 12 labels.
An early stopping mechanism (stop training when the accuracy drops within 2 consecutive epochs) is set up to prevent excessive invalid training in the experiment. The max sequence length is 200, the batch size is 4, the amount of hidden layer is 12, the number of training steps is 500, the number of attention heads is 12, the learning rate is 0.5, and the dropout probability is 0.1. The activation function used is ReLU. The best results on the validation set were obtained with an F1 score of 0.55 ( precision = 0.53, recall = 0.57).
BiLSTM-CRF Model By combining BiLSTM and CRF [
The input of the model is a word sequence, each word in the sentence is expressed as a vector, which should include word embedding and character embedding. And the output is the label predicted by the model for each word, which is a label sequence. The BiLSTM model is used to generate the emission matrix, that is, the probability that each word is marked as a certain label. The emission matrix of the BiLSTM model does not take the constraint relationship between labels into account. For example, in the BIO system, I can not appear after O. So the connection order of tags has to restrict, which will be generated by the CRF model. The CRF model is used to learn the constraint relationship between labels and generate a transition matrix, which can be understood as the probability of connecting another tag behind one tag. When the entire model predicts, it combines the transmission matrix and the transition matrix and uses the V iterbi decoding algorithm to calculate the label sequence with the highest score. The whole process is as follows: the text data is input into the LSTM network, and then into the upper CRF network, and nally, the annotation data is generated.
In this model, the learning rate is 0.001, the batch size is 5, the dropout is
0.5, the optimization algorithm used is Adam. The best results on the validation
set are obtained with an F1 score of 0.56 ( precision = 0.53, recall = 0.61).
The output of the neural network needs to be converted back to Brat's [
The results for each method are shown in Table 1. I provide the results on the development data and the o cially published results on the training data. I have tried to replace the BERT model with the ALBERT model, but the e ect was not as good as BERT, so abandoned this idea. The BioBERT-CRF model was also had been tried, the e ect was similar to BERT-CRF, so it was not been included in the selection.
The best performing version of both the BERT-CRF and the BiLSTM-CRF model is then run on the test set of the eHealthKD corpus. The best score obtained is from the BiLSTM-CRF model is 0.56 of F1. The BERT-CRF model, however, obtained an F1 score of 0.55 on the shared task. Even after downloading my own results, I manually checked the meaning of words with a browser and checked some entities' span. Most of the entity classi cations are ne. I don't know why the score is so low.
All in all, this task is still a long way from being completely completed. Both the problem of the model and the problem of the task need to be solved in the future. 4
Several models were trained and tested in the test set. Using BERT to embed words can bene t the performance of subtask A. In fact, only by using BERT representations and linear dense layers, competitive results can be obtained in the test set. Although the o cial gave us the code to extract the relationship, I still at the rst step and failed to extract all the relationships. This is a very simple step, but this is the rst time have done such a task, and this di culty has not yet been e ectively resolved. Later, due to time constraints, gave up subtask 2.
The nal result only scores 0.17, the possible reasons are as follows: . There is a problem with the evaluation method used locally, which makes the local score look good. . The o cial evaluation result is for English plus Spanish, and I only completed the entity classi cation of Spanish, so the overall e ect is much worse than that in the local area. . The span straddling is realized by myself, and something may have gone wrong.
. The data number is less. 5
This paper describes the participation of team Maoqin in the eHealth-KD challenge at IberLEF 2021. For one method, the pre-trained BERT model has been taken as the basic representation. For another method, BiLSTM has been preferred. The system achieved unsatis ed results in the challenge for o cial evaluation in subtask A. Due to the colab's time and resources limitation, some strategies were not achieved. And subtask B: Relation Extraction has not been completed.
Future work will be done to solve those remainders. More work needs to be
done to improve the performance. Other methods such as transfer learning need
to try. Still, further experimentation is required to understand the impact of the
network's components and how to improve them, which I will explore in future
work. Citing an external corpus may improve the results. The BETO [