The automated analysis of medical documents has grown in research interest in recent years as a consequence of the social relevance of the thematic and the dificulties often encountered with short and very specific documents. In particular, this fervent area of research has stimulated the development of several techniques of automatic document classification, question answering, and name entity recognition (NER). Nevertheless, many open issues must be addressed to obtain results that are satisfactory for a field in which the efectiveness of predictions is a fundamental factor in order not to make mistakes that could compromise people's lives. To this end, we focused on the name entity recognition task from medical documents and, in this work, we will discuss the results we obtained by our hybrid approach. In order to take advantage of the most relevant findings in the field of natural language processing, we decided to focus on deep neural network models. We compared several configurations of our model by varying the transformer architecture, such as BERT, RoBERTa and ELECTRA, until we obtained a conifguration that we considered the best for our goals. The most promising model was used to participate in the SpRadIE task of the annual CLEF (Conference and Labs of the Evaluation Forum). The obtained results are encouraging and can be of reference for future studies on the topic.
Diferent definitions of digital health [
Generally speaking, eHealth approaches could be grouped into two main macro-categories by considering their domain of application: • Wellness: all those applications for fitness and to support a correct lifestyle; • Disease & Treatment Management: all those systems for the management of a pathology, supporting all the phases from diagnosis to treatment and monitoring of the same.
Both these two main kind of eHealth systems,have recently gained a large amount of new
young consumers. eHealth systems and applications are quickly increasing, and the market of
eHealth related systems and mobile application is expected to constantly grow of the 48.44%
each year for the next four years 1. The amount of user of eHalth platforms, which today is
around 825 million worldwide, will cross the billion mark in the next four years. Unfortunately,
despite the increasing amount of data in healthcare available, the percentage of those annotated
and usable for supervised machine learning systems is still very small. As already stated in [
Following we will present some relevant work useful for better understanding the technologies used in the model described in Chapter 3. In Chapter 4, we will discuss the configurations performed and the results obtained, until we present the one that performed best on the data at hand. We will conclude with our considerations and directions for future work.
In recent years, natural language processing has gained a lot of attention, quickly becoming
one of the most trending research areas. In particular, technologies based on deep learning
have made it possible to learn models on a vast amount of data that are highly eficient and
versatile in many scenarios of text analysis and classification. The most famous of such transfer
learning approaches is BERT [
The idea of using BERT as the basis for a name entity recognition task in the medical domain
is not entirely new. The name entity recognition (NER) task consists of identifying n-grams
of a textual content that refers to entities or names relevant for the domain of application. It
is common to find in literature NER systems able to detect in text locations, organizations,
companies, person names but having enough training data it is possible to train those systems
on every entity of interest. The task is a subproblem of information extraction and involves
processing structured and unstructured documents [24]. Since it does not require the annotation
of strictly personal and sensitive data, but generic texts are suficient, this task is widely studied
also in the literature even in the medical domain of application. Mao et al. [25] use BERT and
the Conditional Random Field (CRF) models to identify the name of hospitals, medical staf,
and territory in medical records. Similarly, Xue wt al. [26] fine-tuned the BERT model in order
to detect entities and relations in Chinese medical documents. Vunikili et al. [27] explored
the use of Bidirectional Encoder Representations from Transformers (BERT) based contextual
embeddings trained on general domain Spanish text to extract tumor morphology from clinical
reports written in Spanish. In a similar way, its not dificult to find attempts that use the
RoBERTa model [28], ELECTRA [29] or BioBERT [30]. All those work fall in not evaluating
the eficacy of diferent transformer models before choosing one to use in their models. On the
contrary, we focus on developing a hybrid model that merges an arbitrary transformer based
architecture with a classic CRF layer. In this way, we will not limit ourselves to use with blinded
eyes only a single transformer based model. Still, we will evaluate diferent ones in order to
identify the one that best performing in the medical domain and, in particular, on documents
written in Spanish as provided us by the organizers of the SpRadIe challenge [
The hybrid model we propose is inspired by a common architecture largely used for dealing
with a NER task. In particular, we are looking at the architecture proposed by Huang [31] that is
composed by the concatenation of a Long Short Term Memory (LSTM) model and a Conditional
Random Field layer (CRF) [
The architecture of our proposed hybrid model is showed in Fig. 1. The first step consists of pre-processing the text to make it suitable for the transformer based model. In particular, we split the text into sentences, and we added special tokens at the beginning and end of the sentence. The [CLS] token is used for indicating the beginning of the sentence, [SEP] for indicating its end, and [PAD] tokens are added before the [SEP] for making the input length uniform. For each token, the transformer based model requires input ids, a sequence of integers identifying each input token to its index number in the tokenizer vocabulary provided by the specific pre-trined model. Consequently, the tokens are transformed in their numerical counterpart in order to be properly encoded using token embeddings, sentence embeddings, and positional vectors [33]. The formatted input of length n is denoted as =< 1, 2, .., >, and the corresponding tag sequence is denoted as =< 1, 2, ..., >. The tag annotation schema used for the model is BIO. The schema allows to learn a NER model able to identify tokens in text that can represent one of the following tag: (i) the Beginning (B) of the entity; (ii) the word Inside (I) the entity; (iii) words Outside (O) entities [34].
When the data is ready, it is given as input to the chosen transformer based model. Each one of them has its internal architecture, but all of them allow us to obtain from an embedding representation of each token present in the input sentence. Generally speaking, transformer is an architecture for transforming one sequence into another one by using two modules: the encoder and the decoder. Both encoder and decoder are composed of blocks that can be stacked on top of each other multiple times in order to obtain the desired depth of the network. Each block is mainly composed of a multi-head attention layer, followed by a normalization layer and a feed-forward layer. Using only the encoder module, we can obtain a word embedding representation of each token of the input after any possible arbitrary number of encoding blocks. In particular, BERT, in its basic version, is trained on a Transformer network made of only 12 encoding blocks, 768 dimensional states and 12 heads of attention for a total of 110M of parameters trained on BooksCorpus [35] and Wikipedia English for 1M of steps. The learning phase is performed by scanning the span of text in both directions, from left to right and from right to left, as was already done in Bidirectional LSTM. Moreover, BERT uses a “masked language model”: during the training, random terms are masked in order to be predicted by the net. Jointly, the network is also designed to potentially learn the next span of text from the one given in input. These peculiarities allow BERT to be the current state of the art language understanding model.
By following the logical flow of our hybrid model, we used the embedding generated by the
transformer based model to perform a token-level classification task. In particular, an hidden
layer has been added on top of the stack of the model in order to obtain a prediction matrix
P ∈ R× (i.e. one prediction vector for each on of the tokens) for the given input sequence.
Formally speaking, the classification layer projects each token’s encoded representation to the
tag space R ↦→ R, where is the number of tags which varies in accordance with the number
of classes and the tagging scheme. The CRF layer consequently learns only the transition
probability of the output labels, ∈ R+2× +2 where +2 indicates one tag each for start
and end marker. The matrix is such that , represents the score of transitioning from tag
to tag [36]. For an input sequence =< 1, ..., > and a sequence of tag predictions
=< 1, ..., >, ∈ 1, ..., , the score of the sequence is defined as [
(, ) = ∑︁ ,+1 + ∑︁ P, = = (1) The objective function is the maximum likelihood of the probability distribution denoted as: (2) (3) ⎛ ( (|) ) = (, ) − ⎝∑︁ (,′) ⎞ ⎠ ′∈
We use ⋆ to represent the most likely tag sequence of x that will be considered as output of the model: ⋆ = arg max((, ′))
′∈
During the evaluation, the most likely sequence is obtained by the Viterbi algorithm. The output obtained is re-processed to group back sub-tokens obtained due to the tokenization approach used by transformers models [37]. Only the tag of the first sub-token is considered for the final tag sequence.
The model architecture has been implemented by using the Python programming language. In particular we used the Pytorch [38] syntax and the Transformer Huggingface library [39]. The CRF layer used is the one provided by the library TorchCRF2. We set AdamW as optimizer; a modified version of the Adam optimizer that uses a weight decay factor. Consequently we set a weight decay of 0.01, a learning rate of 1e-5, a number of epochs equal to 500, a number of steps equal to 2000, and a batch size of 16. The code has been run on Google Colab environment3 by using a Tesla T4 Nvidia GPU with 16GB of RAM.
The dataset is provided by the organizers of the SpRadIe 2021 [
The following entities are distinguished: • Anatomical Entity: entities corresponding to an anatomical part, for example breast (pecho), liver (hígado), right thyroid lobe (lóbulo tiroideo derecho); • Finding: a pathological finding or diagnosis, for example: cyst, cyanosis; • Location: it refers to a location in the body. Examples of locations are: walls, cavity, longitudinal, frontal, occipital, cervicodorsolumbosacra, lumbosacral, intracanalar, subcutanea; • Measure: expression indicating a measure; • Type of Measure: expression indicating a type of measure; • Degree: It indicates the degree of a finding or some other property of an entity, for example, “leve”, “levemente” (slight), “mínimo” (minimal); • Abbreviation: acronyms or abbreviations to indicate a medical concept; • Negation: hedge cues indicating negation; • Uncertainty: hedge cues indicating a probability (not a certainty) that some finding may be present in a given patient. 2https://pypi.org/project/TorchCRF/ 3https://colab.research.google.com/
• Conditional Temporal: hedge cues indicating that something occurred in the past or may occur in the future.
In Fig. 2 is reported an example of annotated piece of diagnosis. In Fig. 3 it is showed the distribution of diferent entities among the training dataset. The entity type Finding is particularly challenging, as it presents great variability in its textual forms. It ranges from a single word to more than 10 words in some cases, and comprising all kinds of phrases. However, this is also the most informative type of entity for the potential users of these annotations. Other challenging phenomena are the regular polysemy observed between Anatomical entities and Locations, and the irregular uses of Abbreviations. Moreover the same token can be labeled with diferent tags at the same time making the automatic annotation a complex task.
In order to train the proposed hybrid NER model, we merged the train set with the Same-sample development set resulting in a single set of 222 diagnoses (i.e. 18428 tokens obtained by using a white space splitting). Consequently we used only the Held-out development set for evaluating the performances of the model on 45 diagnoses (i.e. 5090 tokens). The performances of the models are evaluated by using classic metrics such as Precision, Recall and F1-score. As a consequence of the unbalancing of the dataset, we decided to consider the micro average of results obtained for each tag category.
We chose ten diferent transformer models imported through the Transformer HuggingFace library. They are listed in the following: • bert-base-uncased (BERT): it is the standard base version of BERT (12-layer, 768-hidden, 12-heads, 168M parameters) with an uncased English vocabulary; • bert-base-multilingual-uncased (BERT-mul): it is the base version of BERT trained on lower-cased text in the top 102 languages; • dccuchile/bert-base-spanish-wwm-uncased (BETO): BETO is a BERT model trained on a big Spanish corpus. BETO is of size similar to a BERT-base and was trained with the Whole Word Masking technique; • roberta-base (RoBERTa): it is a standard RoBERTa transformers model pretrained on a large corpus of English data in a self-supervised fashion; • xlm-roberta-base (XLM-RoBERTa): it is a multilingual RoBERTa model trained on 100 diferent languages. • google/electra-small-discriminator (ELECTRA): is a new pretraining approach which trains two transformer models: the generator and the discriminator. It has been trained on English data; • skimai/electra-small-spanish (ELECTRA-ES): it is the ELECTRA model trained on Spanish data; • fspanda/electra-medical-small-discriminator (ELECTRA-MED): ELECTRA model trained on English scientific paper in medical domain published on PubMed and PubMEdCentral; • emilyalsentzer/Bio_ClinicalBERT (BioCliBERT): The Bio_ClinicalBERT model is initialized with BERT-Base and it was trained on all notes from MIMIC III, a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA; • seiya/oubiobert-base-uncased (BioBERT): Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University (ouBioBERT) is a language model based on the BERT-Base architecture.
The results we obtained are reported in Table 1-3. It is possible to observe that the best performing model, by considering F1-score as main metric, is BERT in its base version trained on a multilingual dataset. The model has obtained a micro F1-score of 0.69632 that outperforms the competitors in a range from 0.59% to 17.94%. The most significant diference is obtained by comparing the results with them of the ELECTRA-MED base model. In particular, these results are unexpected low and required further investigation for understanding their causes. On the contrary, the model based on BETO is performing particularly well obtaining a results comparable with the one obtained by using BERT multilingual. The diferences among the two models are very low, probably because they share the same internal architecture, and both of them are based on a vocabulary able to deal with Spanish terms. If from one end BERT multilanguage is trained on many diferent languages simultaneously creating some noise on data, on the other end, BETO is trained on a smaller set of data losing then in accuracy. The two issues of the two models are able to produce results somehow very close to each other. If we observe the results obtained for precision (Table 1) and recall (Table 2), we can keep an interesting behavior of the two models. BETO, even if for a small gap, obtains better results for precision than BERT-mul. On the contrary, considering the recall, BERT-mul is the best.
Still looking at the precision values, we can observe that, in general, the chosen models show prediction dificulties for the tag classes "Conditional Temporal", "Uncertainly", "Degree" and "Location". In particular, such behavior could derive from the specificity of such classes for the medical field. It is, in fact, evident as for classes of more general character like "Negation", "Type of Measure", "Abbreviation" all the models perform medium-well. The only exception is the class "Anatomical Entity" that even if much specific of the medical domain turns out to be classified correctly in nearly the totality of the models. This could derive from the wide everyday use that is made of terms regarding the parts of the body also in the common language. Looking at the recall, we can see that some classes were easier to find than others, such as "Anatomical Entity", "Negation", "Abbreviation", "Uncertainly". For the first three classes, obtaining good results of precision and recall, we can say that the models are able not only to be precise in the classification but also to identify them correctly in the text. For the class "Uncertainty", however, the models tend to overestimate its presence in the sentences because it has a good recall value and low precision.
If we want to understand if the training language of the model is an issue for a correct classification of individual tokens, we can observe the diferences between monolingual and multilingual models. Generally speaking, we can observe that multilingual and Spanish-trained models are better performing than their counterpart monolingual. This result was expected as a consequence of the language of the dataset. It is obvious that a model able to manage words in the dataset language is more eficient than a model trained on English data also if the SentencePiece tokenizer, typical of transformer-based architectures, is alleviating these diferences through the use of sub-tokens. If we focus on domain-specific models (BioCliBERT and BioBERT), we can observe that they perform in line with their non-specialized counterparts. The diference in F1-score among BERT-base, BioCliBERT, and BioBERT is only around 0.01. This suggests that the training on the medical domain is not correctly supporting the task of NER for clinical diagnoses, if, as in this case, the model is pre-trained on a diferent language of the testing dataset (i.e. English). In order for it to be possible to draw conclusions about the usefulness of a model trained on medical data for the specific NER task, it would be necessary as a future development to train a new model with medical data in the Spanish language.
By focusing on the diferences in transformer architectures, we can afirm that BERT is commonly performing better than others even if XLM-RoBERTa and ELECTRA are, however, obtaining results not very far from the BERT counterpart. Finally, all the models could not correctly classify Conditional Temporal tags, but this problem is almost certainly related with the size of this class in the dataset. Indeed, only 19 instances are available in the training set and only one instance was present in the set used for the analysis.
In light of the results, decisions were made to submit to the SpRadIe competition. In particular, the first decision taken concerns the configuration of the hybrid model to use. The obvious choice would have fallen on using BERT-mul as the transformer-based model. On the contrary, XLM
Anatomical Entity Conditional Temporal Degree Finding Measure Negation Type of measure Uncertainty Location microAvg RoBERTa was chosen. This decision is motivated by our feeling that the validation dataset was not large enough for containing a representative subset of all the possible variations in data we could have in validation phase, especially regarding the less numerous classes. Indeed, a model best performing on validation set is not always the best performing on test set, especially when the validation set is very small. Therefore, it was decided to opt for a model with intermediate performance in F1-score, following the idea that a less specialized model could better deal the
Anatomical Entity Conditional Temporal Degree Finding Measure Negation Type of measure Uncertainty Location microAvg variability on new unseen data. In particular, we decided to use this strategy as a regularization one, similarly to what is commonly performed with early-stopping for reducing overfitting.
Our second decision concerned the number of models to be learned. In particular, our study of the dataset revealed that the same token could be annotated with more than one tag. For example, the term "cm" could be both parts of a "Measure" tag and an abbreviation tag. To this end, we decided to create four groups of tags with as little overlap as possible and to create
Anatomical Entity Conditional Temporal Degree Finding Measure Negation Type of measure Uncertainty Location microAvg a classifier for each. In particular, the four groups are composed as follows: (i) Finding; (ii) Anatomical Entity, Measure, Degree; (iii) Location, Negation, Type of Measure; (iv) Abbreviation, Uncertainty, Conditional Temporal. In the prediction phase for each tag, we obtained the results of each classifier and chose the one with the highest probability.
If considering the results we obtained from the task organizers reported in Table 4, we can observe that the results are lower than expected, especially for some specific tag class. In particular, "Type of measure" is resulted 116.2% lower than what obtained in the preliminary evaluation. Similar values are obtained also for the classes Uncertainity, Abbreviation and Anatomical Entity. In any case it will be considered future work to investigate the reasons for these poorer than expected performances as soon as the annotated test set is released.
Automated analysis of textual documents in medical settings is still a research challenge that
will be a topic of discussion in the coming years. Recently, the amount of medical data available
has grown rapidly due to the deployment of numerous applications and systems to support the
patient and wellness enthusiast. Nevertheless, the amount of annotated data is still meager due
to their privacy issues. Recently, however, attempts have been made to overcome this problem
with transfer learning strategies based on large pre-trained models that use deep learning
techniques. Among these, those based on transformers have proved to be the most reliable and
versatile. In the wake of these scientific innovations, it was decided to address the problem of
name entity recognition in the medical field by exploiting these approaches. Specifically, we
proposed a hybrid model that combines the power of representation of transformer models
with the predictive power of probabilistic conditional random field models. The model was
evaluated in several of its combinations, varying the transformer model at the core of its
architecture. Specifically, the models BERT, BERT-mul, BETO, RoBERTa, XLM-RoBERTa, ELECTRA,
ELECTRA-ES, ELECTRA-MED, BioBert, BioClinBERT were evaluated. The experimentation
was performed on data provided by the SpRadIE 2021 [
This work has been supported by Apulia Region, Italy through the project ”Un Assistente Dialogante Intelligente per il Monitoraggio Remoto di Pazienti” (Grant n. 10AC8FB6) in the context of ”Research for Innovation - REFIN”. Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URL: https://www.aclweb.org/ anthology/N19-1423. [13] P. Basile, V. Basile, D. Croce, M. Polignano, Overview of the evalita 2018 aspect-based sentiment analysis task (absita), volume 2263, 2018. URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85058658278&partnerID= 40&md5=9719e0512649279a6ed46a8262f050dc, cited By 6. [14] L. de Mattei, G. de Martino, A. Iovine, A. Miaschi, M. Polignano, G. Rambelli, Ate absita @ evalita2020: Overview of the aspect term extraction and aspect-based sentiment analysis task, volume 2765, 2020. URL: https://www.scopus.com/inward/record.uri?eid=2-s2. 0-85097555110&partnerID=40&md5=2122e26ed7367e5a7c40642835591903, cited By 5. [15] K. Florio, V. Basile, M. Polignano, P. Basile, V. Patti, Time of your hate: The challenge of time in hate speech detection on social media, Applied Sciences (Switzerland) 10 (2020). URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85087763770&doi=10.3390% 2fAPP10124180&partnerID=40&md5=d4c6b8ba193ed062299bc19f02cb462e. doi:10.3390/ APP10124180, cited By 7. [16] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, S. R. Bowman, Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint arXiv:1804.07461 (2018). [17] M. Polignano, P. Basile, M. De Gemmis, G. Semeraro, V. Basile, Alberto: Italian bert language understanding model for nlp challenging tasks based on tweets, in: 6th Italian Conference on Computational Linguistics, CLiC-it 2019, volume 2481, CEUR, 2019, pp. 1–6. [18] J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, J. Pérez, Spanish pre-trained bert model and evaluation data, in: PML4DC at ICLR 2020, 2020. [19] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, J. Kang, Biobert: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics 36 (2020) 1234–1240. [20] E. Alsentzer, J. R. Murphy, W. Boag, W.-H. Weng, D. Jin, T. Naumann, M. McDermott,
Publicly available clinical bert embeddings, arXiv preprint arXiv:1904.03323 (2019). [21] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019). [22] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, arXiv preprint arXiv:1911.02116 (2019). [23] K. Clark, M.-T. Luong, Q. V. Le, C. D. Manning, Electra: Pre-training text encoders as discriminators rather than generators, arXiv preprint arXiv:2003.10555 (2020). [24] A. Mansouri, L. S. Afendey, A. Mamat, Named entity recognition approaches, International
Journal of Computer Science and Network Security 8 (2008) 339–344. [25] J. Mao, W. Liu, Hadoken: a bert-crf model for medical document anonymization., in:
IberLEF@ SEPLN, 2019, pp. 720–726. [26] K. Xue, Y. Zhou, Z. Ma, T. Ruan, H. Zhang, P. He, Fine-tuning bert for joint entity and relation extraction in chinese medical text, in: 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE, 2019, pp. 892–897. [27] R. Vunikili, N. SH, G. Marica, O. Farri, Clinical ner using spanish bert embeddings, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020), CEUR Workshop Proceedings, 2020. [28] K. S. Kalyan, S. Sangeetha, Want to identify, extract and normalize adverse drug reactions in tweets? use roberta, arXiv preprint arXiv:2006.16146 (2020). [29] I. B. Ozyurt, On the efectiveness of small, discriminatively pre-trained language representation models for biomedical text mining, in: Proceedings of the First Workshop on Scholarly Document Processing, 2020, pp. 104–112. [30] X. Yu, W. Hu, S. Lu, X. Sun, Z. Yuan, Biobert based named entity recognition in electronic medical record, in: 2019 10th International Conference on Information Technology in Medicine and Education (ITME), IEEE, 2019, pp. 49–52. [31] Z. Huang, W. Xu, K. Yu, Bidirectional lstm-crf models for sequence tagging, arXiv preprint arXiv:1508.01991 (2015). [32] M. Liu, Z. Tu, Z. Wang, X. Xu, Ltp: A new active learning strategy for bert-crf based named entity recognition, arXiv preprint arXiv:2001.02524 (2020). [33] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [34] L. Ratinov, D. Roth, Design challenges and misconceptions in named entity recognition, in: Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), 2009, pp. 147–155. [35] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, S. Fidler, Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 19–27. [36] R. Panchendrarajan, A. Amaresan, Bidirectional lstm-crf for named entity recognition., in:
PACLIC, 2018. [37] T. Kudo, J. Richardson, Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, arXiv preprint arXiv:1808.06226 (2018). [38] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., Pytorch: An imperative style, high-performance deep learning library, arXiv preprint arXiv:1912.01703 (2019). [39] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al., Huggingface’s transformers: State-of-the-art natural language processing, arXiv preprint arXiv:1910.03771 (2019).