This paper presents our participation on the BioASQ tasks 8a and MESINESP. Our approach was based on X-BERT, a state-of-theart deep learning algorithm developed for Extreme Multi-Label Classi cation (XMLC). We adapted this algorithm and its most recent version to the biomedical and multilingual biomedical panorama which, to the best of our knowledge, is the rst XMLC-solution to be applied to both tasks. In addition, we combined this algorithm with a named-entity recognition tool to nd entities from DeCS and MeSH in the text in order to improve the assignment of labels by the XMLC algorithm. In both challenges, our results in the Micro F-Measure, which was the most relevant evaluation measure, were low. However, our submissions achieved top results in the precision measures, reaching the 1st place in the last two weeks of task 8a in the Micro Precision measure and 2nd place in MESINESP in Micro and Macro Precision, as well as Example Based Precision.
BioASQ is a series of international challenges focused in promoting the development of state-of-the-art solutions for NLP tasks, namely semantic indexing and question answering, for the biomedical domain. This year, in addition to the usual annual competitions, BioASQ presented a new task called MESINESP (Medical Semantic Indexing in Spanish) focused on the multilingual panorama, in this case, the Spanish language. In this paper, we describe the approach of our submissions for both BioASQ task 8a and BioASQ MESINESP task.
The BioASQ task 8a consisted in indexing English biomedical abstracts with terms from MeSH (Medical Subject Headings) related with the content of biomedical abstracts retrieved from PubMed. The BioASQ MESINESP task was similar, but instead of using MeSH, it used DeCS (Health Sciences Descriptors), a hierarchy of terms closely related to MeSH that can be used to label biomedical articles in Spanish, Portuguese and English. The biomedical abstracts to index were in Spanish and were retrieved from the IBECS and LILACS databases.
In both competitions, each abstract could have more than one term indexed
to it, making this a multi-label classi cation challenge. However, since there are
more than 29,000 MeSH terms and more than 33,000 DeCS terms to choose from,
our approach consisted in using an Extreme Multi-Label Classi cation (XMLC)
algorithm to classify the abstracts complemented by a Named Entity Recognition
tool. XMLC algorithms can provide the best subset of labels to index data,
choosing the labels from an extremely large label set, which can reach hundreds
of thousands or even millions of labels [
We have chosen the XMLC algorithm X-BERT [
In addition, the algorithm was also combined with MER (Minimal
NamedEntity Recognition) [
Recently, many deep learning solutions developed for NLP tasks use pre-trained
language models in their core, such as BERT (Bidirectional Encoder
Representations for Transformers) [
Pre-trained language models also constitute an important part of the resources
available to develop deep learning models for multilingual NLP task. BERT has
also been applied to the multilingual panorama through three di erent models:
BERT Base Multilingual Cased, BERT Base Multilingual Uncased and BERT
Base Chinese. The larger model contains 104 languages, and it is composed
by the text of the Wikipedia pages for each of those languages, excluding the
user and talk pages. These BERT models surpassed the highest accuracy scores
achieved in the XNLI dataset [
Another great source of multilingual pre-trained models is the Hugging Face
Transformers library1 [
Finally, another example of multilingual improvement, namely in the
biomedical domain, is a recent work that aimed at the development of word embeddings
for the Spanish biomedical domain [
As to XMLC, several machine learning solutions have been developed in the last
decade [
architecture was successful in applying deep learning to XMLC, surpassing most state-of-the-art algorithms in several datasets.
Another successful approach was AttentionXML [
Lastly, one of the most recent approaches is X-BERT [
More recently, a newer version of X-BERT has been released, renamed
XTransformer2[
For both tasks, modi cations in the algorithm code were required. The rst one was made in the vectorization of the labels of the training, test and validation sets. We have chosen to use all possible labels, including the labels that were not present in the train, test or validation sets. This change was needed since the algorithm would fail to work correctly if the number of labels between sets did not match. Another modi cation was the inclusion of SciBERT in the choices of models to train X-BERT, so that we could use this model in task 8a. However, this inclusion was not made in X-Transformer, since we only used X-Transformer for the MESINESP task.
X-Transformer required additional changes since, at the time of the
competition, the algorithm was still being developed by the X-BERT team and it
could only process the four datasets in which X-BERT was tested [
Finally, in addition to these modi cations, both X-BERT and X-Transformer were also changed so that the algorithms could process input data containing diacritical marks, such as accents, that are common in the Spanish language. 3.2
A common pipeline was developed for both tasks, with some changes according to the algorithm used or to the task, as it can be seen in Figures 1 and 2. The rst step, was to retrieve the data given by the competition organizers. A total of 318,658 articles abstracts and titles were used for both task 8a and MESINESP for comparison purposes. The articles were then split into training, test and validation sets using a 60%-20%-20% proportion to be used by XBERT. The validation set would be used for hyperparameter tuning such as the number of training epochs, since X-BERT can perform an evaluation during training and then save the hyperparameters in a checkpoint if the evaluation results surpassed the results of the previous checkpoint. The values di er when using X-Transformer, because it only requires training and test sets. The test set will also be used by X-Transformer to save the model checkpoints during training evaluation. Therefore, we decided to use a proportion of 70%-30% for X-Transformer. A summary of this distribution can be seen in Table 1.
The next step was the creation of a vocabulary le, which is required by both X-BERT and X-Transformer and that contains the labels used to classify the data and a corresponding numerical identi er. For that, each line of the vocabulary le has a DeCS or MeSH code, according to the task, and its corresponding internal identi er, which corresponds to a number from 0 to N, where N is the total number of DeCS or MeSH Terms minus 1. This internal numerical identi er is the characteristic that allows X-BERT and X-Transformer to work with any type of labels and with any kind of language, since the algorithms will use these numeric identi ers to classify the text. In addition, a label mapping le was created for each task. For MESINESP, it contains the correspondence between the DeCS term, its code and its numeric identi er in the vocabulary le, while for task 8a, the le has the same structure but using the MeSH terms instead. For example, the term `Temefos', which has the corresponding DeCS code `2', is the rst element in the vocabulary le, thus its numeric identi er will be `0'. This label mapping le will later be used to map the predictions from their numeric identi ers to the corresponding DeCS or MeSH codes required for the task.
After converting the codes to numeric identi ers, we used MER [
We chose to train some models using the abstracts and others using the titles in order to compare the performance of the algorithm according to the amount of text given in each article. In cases where the title for a speci c article was not present, we used its abstract instead, and in cases where the abstract of a speci c article was not present, we used its title instead. In the end, the les to be given as input to X-BERT or X-Transformer, contained, for each article, the list of corresponding DeCS or MeSH terms indexed to that article converted to their corresponding internal identi er, the article's title or abstract, and the terms identi ed by MER.
The results of X-BERT and X-Transformer are given in the form of sparse matrices, with a number of rows equal to the number of articles that compose the test set, and the columns corresponding to the possible labels. The prediction for each article is retrieved, comprising a top K of most relevant labels and their corresponding con dence values. We used K=20 labels per article for both XBERT and X-Transformer. For a prediction to be chosen as correct, we discarded every label with a con dence value under a threshold, which will be more detailed in Section 4. Each label is then converted to the corresponding DeCS or MeSH term by using the previously created label mapping le, so that in the end,
the JSON le containing the predictions has the MeSH or DeCS codes for each article id.
In the test sets given by competition organizers, there were no MeSH or DeCS terms indexing the articles, so we had to put a placeholder label on each article, because the code was not prepared to run on unlabeled data. We also had to arti cially adapt the size of the given test sets by adding extra articles to them, so that they could have the same size as the ones used on the trained X-BERT or X-Transformer models, otherwise the algorithm would fail to work. In this case, the set needed to have a total of 63,732 articles for X-BERT models and 95,598 articles for X-Transformer models, which corresponds to the number of articles used in the test set of those models. Since in both competitions the number of articles given to classify was under that value, we padded the test set with additional articles equal to the di erence. For task 8a, we included indexed PubMed articles from the 14 million articles given by the competition as training data. For MESINESP, we included translated and indexed PubMed articles that came from a larger dataset given as additional material by the task organizers to the participants. The additional articles in both tasks were used as an additional validation set to de ne the con dence threshold values of our submissions and were not used in neither set to train nor evaluate the models, so that the results would not come biased. 4
To evaluate the results achieved by our models, we used a script developed by us to measure the Macro and Micro Precision, Recall and F1 scores, which were some of the measures used in both BioASQ competitions. For each model, two evaluations were made: one using the test set, and another using the additional validation set composed by the articles that were padded to the test sets given by the competition organizers which acted like a preliminary evaluation for the competition.
The evaluation script was focused on choosing the con dence score threshold
that achieved the highest Micro F1 (MiF) score and the highest Micro Precision
(MiP). The focus on MiF was chosen, since MiF was de ned by the
organization as the most important measure to both challenges. The focus on MiP was
motivated by the improved precision of X-BERT over other XMLC algorithms
[
The con dence score interval given by X-BERT to its predicted labels ranged from 0 to 1. However, in X-Transformer, we noticed that this interval was changed from 0 to 1 to -0.99 to 1, thus encompassing negative con dence values. To facilitate the comparisons between X-Transformer and X-BERT, we have normalized this scale to t the interval from 0 to 1. Then, to determine the best threshold for the con dence score that achieved the highest MiF or MiP values, we tested the values between 0.01 and 1, incrementing by 0.01.
Given the token limit of 512 tokens of the used BERT models, we considered using only the articles or the titles to train the models since that, if we combined both articles and titles along with the entities recognized by MER, in most cases the limit of 512 tokens would be surpassed, thus the articles would be truncated and consequently relevant information and the list of entities found by MER would be lost. Therefore we decided to check which combination could achieve better results: if using the titles and the MER terms or if using the abstracts and the MER terms.
The main hypothesis was that, due to the increased objectiveness of the titles in the core topics of the article in a less amount of words than the abstract, along with a list of key terms and synonyms given by MER, the model would be able to achieve better results since it a had a continuous string of words that were more related to the topics of the article than a larger amount of words with the key topics more diluted. To test the hypothesis, we used X-BERT using the BERT Base Multilingual Cased model and the MESINESP dataset, which was made available rst by the organizers. The results are available in Table 2.
Analyzing the results of this experiment, we can see that the di erence between using the titles and the abstracts is minimal. If evaluation was focused on achieving the highest MiF value, the model that used the abstracts achieved slightly higher MiP and MiF values. However, if the evaluation was focused on attaining the highest MiP value, the model using the titles achieved better scores in all measures than its abstract counterpart, which con rms our hypothesis.
Titles + MER
Focus
Measures
MiP MiR MiF MaP MaR MaF MiF 0.4780 0.4123 0.4427 0.4766 0.4094 0.4028 MiP 0.7117 0.1574 0.2578 0.6812 0.1642 0.2480 MiF 0.4685 0.4142 0.4397 0.4704 0.4229 0.4072 MiP 0.7154 0.1636 0.2663 0.6901 0.1769 0.2617 Threshold 0.10 0.93 0.11 0.92 5.2
Only two models were trained for the BioASQ task 8a with the following characteristics: { Model 1: MER in conjunction with X-BERT, using BERT Base Uncased and netuned using article's titles. { Model 2: MER in conjunction with X-BERT, using SciBERT and netuned using the article's titles.
Both models used the titles due to results shown in the previous section and due to time restrictions, since the competition had already begun when we started training the models, so we could only train 2 models. Therefore, Model 1 was trained for Batch 2 of the competition, while Model 2 was trained for Batch 3.
For both models, the parameters given as default to train and evaluate were kept, except for the eval and train batch sizes which were changed from their corresponding default values of 64 and 32 to 3 due to hardware limitations. The number of train epochs was also changed to 7, since the best checkpoint would usually be achieved before this epoch. The models were trained on a single NVIDIA Tesla P4 GPU, taking 1 week for each model to be fully trained. 5.3
In total, four models were trained for the BioASQ MESINESP task with the following characteristics: { Model 1: MER in conjunction with X-BERT, using BERT Multilingual Base
Cased and netuned using the article's abstracts. { Model 2: MER in conjunction with X-BERT, using BERT Multilingual Base
Cased and netuned using the article's titles. { Model 3: X-Transformer using BERT Multilingual Base Cased and netuned using the article's abstracts. { Model 4: X-Transformer using BERT Multilingual Base Cased and netuned using the article's titles.
From these four models, we decided to train two using X-BERT and the other two using X-Transformer since it was made available during the time of this competition. We also decided to train two models using the abstracts and the other using the titles, so that we could not only compare the di erences between the two versions of the algorithm, but also to check if our hypothesis from Section 5.1 would also be con rmed in the results achieved in the competition.
For all models, we have kept the default parameters to train and evaluate the models, except for the eval and train batch sizes, due to hardware limitations. In X-BERT they were both changed from their corresponding default values of 64 and 32 to 3. In X-Transformer, both were changed to 4 and we also set the number of gradient accumulation steps to 2 to compensate for the small batch size. The X-BERT models were trained on 7 epochs, since the best checkpoint would usually be achieved before this epoch, like in the BioASQ task 8a models. The X-Transformer models were trained during 3 epochs, the default value. All models were trained on a set of 4 GPUs composed by 1 NVIDIA Tesla P4 GPU and 3 NVIDIA Tesla M10 GPUs. Each model took approximately 4 to 5 days to be fully trained.
The results of the preliminary evaluation using the models test sets can be seen in Table 3. When focused on MiF, both Model 1 and Model 2 achieved similar results for their best scoring threshold values, which was the same. However Model 2, the one trained using SciBERT, achieved slightly higher values, which was expected since SciBERT was especially created to be used with scienti c text. MiP and Macro Precision (MaP) have the highest values, and both Recall and F1 scores have a relatively similar score and are both higher than 0.50.
When focused on MiP, the threshold values di er, but the results of both models were again similar, with Model 2 achieving slightly higher values in all measures, except for the Macro Recall, which was 0.01 lower compared to Model 1. Both MaP and MiP increased signi cantly to values over 0.83 and reaching a maximum of 0.89. However, this gain in precision comes with the cost of the recall and F1 values dropping between 0.17 and 0.27. We competed in both batch 2 and 3 of BioASQ Task 8a. The results achieved in each week of the batch can be found on Tables 4 and 5, where our scores are compared with the solution that achieved the highest MiF score on the same week.
In batch 2, we submitted results for weeks 3, 4 and 5. The submitted solutions were given by Model 1, the model trained using BERT Base uncased, with the con dence score threshold that achieved the highest MiF score. The results in those weeks were suboptimal, with our model staying in the last position every week and in all evaluation measures.
For batch 3, we submitted results from weeks 2 to 5 using Model 2. In week 2, we used the same strategy of the previous batch, focusing of achieving the highest MiF score. However, the results were very similar, with only a small increase in some precision measures. From week 3 an onward, we decided to started submitting the predictions focusing on the highest MiP value, since it was our best scoring measure in the previous submissions as well as our best scoring measure in the preliminary evaluation. As a result, we achieved lower accuracy, recall and MiF scores, although the loss in MiF was less than 0.1. However, all precision measures increased signi cantly making the model reach the 3rd, 1st and again 1st place in the MiP measure on weeks 3, 4 and 5, respectively, and a 1st place in the Example Based Precision (EBP) in the nal week. The only precision measures where the model did not have a signi cant increase were the MaP and the Lowest Common Ancestor Precision (LCA-P) measures. 7 7.1
Contrarily to the task 8a preliminary evaluation, for task MESINESP we added a new evaluation parameter, which was the con dence score threshold of 0.50. We decided to add this value since it was the middle of the con dence interval scale and it should provide a baseline performance.
The results achieved in the rst evaluation using the test set can be seen in Table 6. For both Model 1 and Model 2, the scores achieved by each model were very similar, with slightly higher scores achieved by Model 2. As for Models 3 and 4, we noticed that they achieved higher values than the ones achieved by the older version of the algorithm. Also, and contrarily to Models 1 and 2, Model 3, which used the abstracts, achieved higher values in all measures when compared with Model 4, except when focused on MiP, where Model 4 achieved higher Precision values.
However, in the second evaluation which used the additional validation set, the results achieved were di erent as can be seen in Table 7. First, there was a drop in the scores, especially in the recall measures thus dropping the F-score. When focusing on MiP, the best performing model was Model 2 and not Model 4 as it was in the rst evaluation. This small di erence might be due to the usage of MER in Model 2, while Model 4 did not. As to the focus on MiF, the best scoring model was Model 4 with the highest Macro and Micro recall and F-scores.
After this second evaluation, we have decided to submit a total of 4 prediction les, two from a X-BERT model and the other two from a X-Transformer model so that the performance of the models could be compared in the competition. The results of Model 2 focusing on MiF and MiP were chosen, since the focus on MiP presented the highest precision scores, and the focus on MiF had better recall and F-scores than Model 1. From Model 4, the results focusing on MiF and the baseline were chosen, since the focus on MiF had the achieved the highest recall and F1 score values, and the baseline since the precision gains from the MiP focus were not so signi cant and had heavier losses on the remaining measures. In Table 8, we present the results achieved by our four submissions, the BioASQ baseline and the winning system. Like in BioASQ task 8a, our results in the MiF measure were not optimal, with the four submissions staying in bottom positions in the classi cation. However, in the precision measures, the scores achieved by two of our submissions were higher than most systems, even surpassing the winning system.
The predictions of Model 4 focused on the baseline achieved the 2nd place in the MiP, MaP and EBP measures, while the predictions of Model 2 focused on MiP achieved the 3rd place in MiP as well as a 5th place in EBP. As to the models focused of MiF, they achieved balanced scores between all measures, but they were not enough to surpass the BioASQ baseline. We can also notice that the submissions focused on MiF achieved higher accuracy scores, which was expected. LasigeBioTM TXMC F1 0.2507 0.3559 0.1936 0.0858 0.3646 0.0799 0.1440 0.3641 0.1986 0.2380 (Model 4 focus MiF) (LMasoidgeelB4iofTocMusTbXaMselCinPe) 0.1271 0.6864 0.0701 0.0104 0.6989 0.0081 0.0708 0.6609 0.0716 0.1261 8 8.1
Overall, the results achieved in both competitions showed that the usage of deep learning XMLC solutions in the biomedical and multilingual panorama can achieve promising results, especially with the high MiP values achieved in MESINESP and in the last three weeks of task 8a. However, there is also a great room for improvement. For example, in both task 8a and in MESINESP, the models struggled in all recall measures, with scores lower than most of the competing systems, although achieving higher scores in the precision measures. We think that these low recall values can possibly be explained by several factors.
The amount of data used to train the models could have impacted our results. We have chosen to use the same number of articles in both competitions, which was the number of available articles in the pre-processed train set from task MESINESP, so that we could compare the performance of a biomedical XLMC based model across multiple languages. There was much more data available for both task 8a and task MESINESP that we could have used to train larger models. However, this was not possible to do in time due to the hardware limitations, which lead to each model requiring approximately 1 week to train.
Another limitation was the fact that we did not use more complex lexicons in MER to identify entities in the text. The lexicon we used in task 8a was rather simple, only comprising the MeSH terms and their synonyms. We could have used more complex lexicons such as one with the term's ancestors or with ontology information, which could possibly increase the nal scores. In task MESINESP, the fact that we did not use MER in the X-Transformer models could have negatively in uenced our results.
Finally, another reason for our low recall values could be caused by the fact that XMLC algorithms tend to achieve higher precision values due to an increased focus on precision measures, namely in Precision at top K (P@K). P@K is widely used in the evaluation and comparison between XMLC algorithms [1{5, 15, 16], as it considers the number of most relevant labels retrieved by the algorithm among the top K documents. The choice of using this measure to evaluate the performance of these algorithms makes sense when dealing with datasets with thousands of documents, from which the objective is to retrieve the most relevant labels from a set of thousands or millions of possible labels. However, this increased focus in precision can a ect negatively the evaluation of the systems in other measures, such as the F1 score and recall, which were evaluated in these two tasks. 8.2
One of our objectives with our participation in these two tasks was to compare the performance of an English model versus its multilingual counterpart in a biomedical domain. Comparing the results achieved by our submissions in BioASQ task 8a and in task MESINESP, we notice that there is a great di erence between the performances of an English model and a multilingual model. In task 8a, Model 1 and 2 shared the same characteristics as task MESINESP Model 2. The three of them were X-BERT models trained with the same number of articles, all used the titles and were combined with MER to nd key terms in the abstracts. The major di erence between them was the pre-trained language models used. In task 8a, the models used BERT Base Uncased and SciBERT, while in task MESINESP the model used BERT Base Multilingual Cased.
Analyzing the scores achieved by these three models, we can notice that there is a signi cant di erence in the scores achieved in both our evaluations and in the competition evaluations. The clearest is that in task 8a evaluation, our worst scores in the competition surpass our best submission in task MESINESP in the non-precision measures, as well as the enormous di erence in the scores between the rst classi ed systems in task 8a and the MESINESP winner. The reason for this di erence can be caused by the number of documents to classify in each task, which in MESINESP was more than 24,000 articles in total, whereas in task 8a it was about 5,000 to 7,000 articles in each week.
However, in our evaluations using the model's test set, we can also see that
the English-based models achieve better results in all measures, as can be seen in
Table 9. The reason for the di erence in the results of the evaluations is probably
caused by the amount of data used to train the pre-trained language models.
The BERT models were trained using mostly text data retrieved from Wikipedia.
However, the amount of Wikipedia text varies between di erent languages. For
example, in terms of words, the BERT Base model was trained with more than
3,000 million words, with about 2,500 million coming from the English Wikipedia
[
We could also have developed a model using BioBERT [19], a BERT model developed with English biomedical text for NLP tasks, but due to time and hardware constrains, we were not able to adapt and train a model using BioBERT, which could have lead to improved results in task 8a. In the future, we intend to develop a X-BERT and a X-Transformer model using BioBERT and compare its performance with the models trained using SciBERT.
Finally, as a way of further improving the results, we could use semantic similarity solutions to nd which labels are more related with the abstract, thus hoping to reduce the number of false negatives and consequently improve the recall and F-measures. 9
With our participation in these two competitions, we successfully adapted and applied a state-of-the-art deep learning XMLC solution to the biomedical and multilingual biomedical panorama. As far as we know, deep learning XMLCbased solutions had not yet been applied to both these scenarios. The results achieved in the competitions were promising, with our submissions achieving top classi cations in the precision measures when compared with most competing models. Although our results were not optimal, we achieved a 1st place in MiP in the last two weeks of task 8a and 2nd place in MiP, MaP and EBP in task MESINESP. However, the performance of multilingual models is still far from the performance of English models in NLP tasks, especially in the biomedical domain. Therefore, we can conclude that there is a great need for additional research and development of non-English deep learning models and corpora, especially for speci c domains such as the biomedical sciences. 10
This project was supported by FCT through funding of the DeST: Deep Semantic Tagger project, ref. PTDC/CCI-BIO/28685/2017, and the LASIGE Research Unit, ref. UIDB/00408/2020 17. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanovand, V.: RoBERTa: A Robustly Optimized BERT Pretraining Approach. Retrieved from: http://arxiv.org/abs/1907.11692 (2019) 18. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V.: XLNet: Generalized Autoregressive Pretraining for Language Understanding. Retrieved from: http://arxiv.org/abs/1906.08237 (2019) 19. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., and Kang, J.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics 36 (4), pp. 1234|1240. (2020) https://doi.org/10.1093/bioinformatics/btz682