MeSH (Medical Subject Headings) is a large thesaurus created by the National Library of Medicine and used for fine-grained indexing of publications in the biomedical domain. In the context of the COVID-19 pandemic, MeSH descriptors have emerged in relation to articles published on the corresponding topic. Zero-shot classification is an adequate response for timely labeling of the stream of papers with MeSH categories. In this work, we hypothesise that rich semantic information available in MeSH has potential to improve BioBERT representations and make them more suitable for zero-shot/few-shot tasks. We frame the problem as determining if MeSH term definitions, concatenated with paper abstracts are valid instances or not, and leverage multi-task learning to induce the MeSH hierarchy in the representations thanks to a seq2seq task. Results establish a baseline on the MedLine and LitCovid datasets, and probing shows that the resulting representations convey the hierarchical relations present in MeSH.
COVID-19 Pandemic
With the outbreak of the COVID-19 disease, the
biomedical domain has evolved: new concepts have emerged, and
old ones have been revised. In that context, scientific
papers are typically manually or automatically labelled with
MeSH terms, Medical Subject Headings [
Another obstacle (independent from the pandemic) is
the scale of the MeSH thesaurus, as there are thousand
of MeSH descriptors. State-of-the-art on MeSH
classiifcation thus uses IR techniques [
In this work we rely on BioBERT [4] to extract representations from paper abstracts and classify them. Such model is pretrained with masked language modeling objectives on data from the biomedical domain, and we assume that BioBERT encodes some semantic knowledge related to the biomedical domain. However, it has been shown that this pretraining might not be optimal for tasks such as NER or NLI [5].
We formulate the zero-shot task as an “open input” problem where we take both the class and the text as an input, and output a matching score between the two.
The motivation behind this formulation is that the model
efit from very large datasets they can be trained on in
a self-supervised way. Such pretraining allows them
to learn rich semantic representation of the text, and
perform knowledge transfer on other lower-resourced
tasks. Those pre-trained models can be used in a
zeroshot setting, by creating representations for the given
document and each of the the diferent classes, and then
computing similarity scores based on those
representations. Chalkidis et al. [9] proposed for example to
compute the similarity score with an attention mechanism
between classes and documents representations. Rios and
Kavuluru [10] proposed in addition to the attention
mechanism to include hierarchical knowledge using GCNN,
but they do not handle the case where the hierarchy is
only available during training. Wohlwend et al. [11] also
worked on the representation space using prototypical
network and hyperbolic distances and they showed that
there was still possible improvements in metric learning.
would learn to use the semantics of the class labels, and
thus will be able to extend the semantic knowledge
encoded by pretrained model (eg. BioBERT) to new classes. Fine-grained biomedical classification. BioASQ
Therefore, our assumption is that those models host and challenge is one of the reference on fine-grained
clascan make use of a good representation of the semantics sification of biomedical articles; however the challenge
underlying the MeSH hierarchy, including unobserved does not focus on zero-shot adaptation, which is the
sceterms. nario we consider in this work. Mylonas et al. [
In order to improve the semantic representations of tried to perform zero-shot classification across MeSH pretrained model, we propose a multi-task training frame- descriptors, but their testing settings considered only work, where an additional decoder block predicts the a small number of MeSH descriptors. In our work we position in the hierarchy of the input MeSH term. MeSH try to perform a larger scale evaluation in context of descriptors have a position in the hierarchy that is de- the pandemic. Finally, [12] proposed an architecture for ifned by their Tree Numbers (see Figure 2), so the goal hierarchical classification tasks that is able to learn the of this secondary module would be to generate those hierarchy by generating the sequence from the hierarchy Tree Numbers during training. Model learnt with this tree (using an encoder/decoder architecture). Our work additional task should better encode MeSH hierarchical considers similar architecture in a zero-shot scenario. structure and hopefully improve zero-shot and few-shot capacities of thus learnt representations. Enforcing that Probing. Probing models [13, 14] are lightweight classemantic knowledge is embedded in the model also guar- sifiers plugged on top of pretrained representations. They antees a degree of explainability, an important feature in allow to assess the amount of “knowledge” encoded in the the medical domain. pretrained representations. Alghanmi et al. [15], Jin et al.
The main findings of the work are: (a) Our multi-task [5] introduced frameworks in the biomedical domain for framework improves precision on some datasets, and disease knowledge evaluation focusing on relation types thus the F1-score, but it is not systematic. (b) Probing (Symptom-Disease, Test-Disease, etc.), while we are willtasks show that performance increases are directly linked ing to assess how well a hierarchical structure is encoded to a better knowledge of the MeSH hierarchy. Still, in- in representations. We rely on the structural probing cluding hierarchical information on a large scale dataset framework [16] that we compare against the hierarchical (Medline) is dificult, especially in few-shot and zero-shot structure encoded by MeSH thesaurus. settings.
on zero-shot learning, which consists in classifying samples with labels not seen in training [6, 7]. Pretrained models such as BERT [8] or BioBERT [4] are central to zero-shot learning in NLP. These models ben
First, we will explain the architectures we explored to address the zero-shot classification problem, and more precisely the multi-task learning framework. Then, we will focus on the design of the probing tasks, that we used to analyse to what extent the hierarchical knowledge was encoded by the diferent representations. architecture is similar to the BioBERT STL, but in ad- in the models parameters [17, 18], to allow the model where both 1 and 2 are learnable parameters included
The first model is a BioBERT encoder, followed by a dense layer on the [ C L S ] token. Input of BioBERT is composed of the MeSH term, the MeSH description and a document abstract: [ C L S ] M e S H t e r m : M e S H d e s c r i p t i o n [ S E P ] A b s t r a c t , and output is a single neuron, that goes through a sigmoid activation function.
As an example, input for the MeSH term Infections is [ C L S ] I n f e c t i o n s : I n v a s i o n o f t h e h o s t o r g a n i s m b y m i c r o o r g a n i s m s o r t h e i r t o x i n s o r b y p a r a s i t e s t h a t c a n c a u s e p a t h o l o g i c a l c o n d i t i o n s o r d i s e a s e s . [ S E P ] A b s t r a c t . dition to the binary classification task it learns simultaneously an additional task of MeSH term hierarchical position generation. The motivation behind this additional task is that the learnt representations would better encode hierarchy of MeSH terms and hopefully better deal with zero-shot classification or fine-grained classification problems. _ = _ℎ × ℎ ) = ̂
+ ℎ+1 , +1 = (ℎ × _ , _ℎ ) (1) (2) (3) (4) (5) where
_ℎ is the output of BERT, of shape (512, 768), ℎ the hidden state of the GRU cell, of shape (768,), and the embedding of the current word, also of shape (768,). On line (4), the + operator corresponds to a sum of the two vectors (both of shape (768,)) in each of their dimensions. For the word generation, through a dense layer and a logsoftmax function. +1 is then passed Note that
_ℎ is formed from the output tokens of BioBERT corresponding to the MeSH description (by applying the MASK to all other tokens). We also apply “teacher forcing”, to reduce error accumulation.
The original problem is thus transformed in a multitasks problem, where the two losses (binary cross entropy and negative log likelihood losses) are then jointly learned: = 1 2 12 1 + 1 2 22 2 + log( 1 2) (6) to balance between the binary and tree number generation losses. The last regularization term is only here to prevent the model to learn the naive solution of just increasing 1 and 2 to reduce the loss.
The output vocabulary of the decoder is composed of the tree numbers tokens: A - Z letters, 0 0 - 9 9 digits, and 0 0 0 9 9 9 digits. All together, the vocabulary size is around 1100, on which we apply an embedding layer to transform discrete tree numbers to a continuous embedding space. As this vocabulary is completely new, embedding is learnt from scratch using back-propagation. Note that for MeSH descriptors that have multiple tree numbers, we just duplicate the inputs to learn the multiple positions in the hierarchy.
In both architectures, the full set of parameters is updated during training. Thus, each model provides new representations of the input text and labels, that are further evaluated through zero-shot classification or through probing.
To better understand the capacity of pretrained representations to encode hierarchical relations of biomedical terms, we considered two probing tasks, adapted from [16]. The objective is to test whether the representations learned by the model are linearly separable with regards to the probing tasks.
We define the probing task from a main model, taking as input a MeSH descriptor m and returning its internal representation h . We then recall that it is possible to define a scalar product definite symmetric matrix ∈ from any matrix ∈
R×
h h from any positive semi
S+× , and more generally by taking = . Using the metric distance corresponding to this scalar product we then define a distance from any matrix :
(h , h ) = (( h − h )) (( h − h )) with h and h the representations of two MeSH descriptors (more details on representations in section 5.2). Our model has as parameter the matrix , which is trained to reconstruct a gold distance from one MeSH term to another. More specifically, the task aims to approximate by gradient descent : , ∑ | (h , h ) − (h , h )2| tance. We also note that, as in the original paper, we add a squared term on the predicted distance. Concerning the dimensions of and , is the dimension of the representation space (same than h), and is the dimension of the linear transformation which we will take equal to 512. We did not further experiment on the dimension of the linear transformation (see the original paper by [16] for discussion on both the squared distance and the linear transformation dimension).
The only diference with the original paper is the definition of the gold distance. We have evaluated two probes: In this section we present the datasets we used to train the models and evaluate the corresponding representations. We also explain how we construct a zero-shot dataset out of the Medline dataset with MeSH annotations.
Medline is the US National Library of Medical/Biomedical dataset2, containing millions of biomedical scientific documents, and built around the MeSH thesaurus (Medical Subject Heading). This thesaurus contains about 30,000 MeSH descriptors, updated every year, and used for semantic indexing of the documents. These MeSH terms also define a hierarchy: the first level separates the MeSH terms into 16 main branches, then each MeSH term is the child of another more general MeSH term, and this over up to a depth of fiteen. The sequence of nodes traversed to reach a MeSH term from the root is called Tree Number.
An example of the hierarchy is shown in Figure 2 with the two MeSH COVID-19 and Bronchitis. For example here, Covid-19 has the Tree Number C 0 1 . 7 4 8 . 2 1 4 (C beC 0 1 for Infections, C 0 1 . 7 4 8 for Respiratory Tract Infections, then C 0 1 . 7 4 8 . 2 1 4 for Covid-19).
The majority of MeSH descriptors from the hierarchy have multiple Tree Numbers, so the hierarchy follows a directed acyclic graph structure. Also, the annotation of scientific documents with ancestors of a MeSH is not always explicit. For example, a document can be indexed with term Covid-19, but not necessarily with terms Infections or Respiratory Tract Infections.
There are on average 13 annotated MeSH descriptors per document, where 2 or 3 will be annotated as major MeSH to indicate that the document deals more specifically with these topics. In our work, we use the whole tion to the MeSH annotation and hierarchy, the Medline database provides a description for each MeSH term, used by our models as specified in section 3.1.
LitCovid is a subset of the Medline database [19, 20], where extraction is done via PubMed (search engine of Medline), using the keywords: “coronavirus”, “ncov”, “cov”, “2019-nCoV”, “COVID-19” and “SARS-CoV2”. Using this subset of articles allow us to work more specifically on COVID-19 related articles, with also a subset of 9,000 COVID-19 related MeSH descriptors (instead of the full set of MeSH descriptors). The LitCovid dataset dicting the number of common ancestors given two MeSH descriptors, but the regressor was unable to train from the representations, hence the use of binary tasks. where is the predicted distance and the gold dis- ing the main branch Disease, then we have 3 sub-levels: 1. Shortest-Path Probe: given two MeSH descrip- set of major and non-major MeSH descriptors. In additors, we ask the model to predict the distance between the two MeSH terms, as the length of the shortest path in the graph defining the MeSH hierarchy; 2. Common-Ancestors Probe: model predicts positive)1. whether two MeSH descriptors have k common ancestors. For this second task we thus define multiples binary probe models that predict if the two MeSH terms have at least k common ancestors or not (for k between 1 and 3). In this particular case of a binary probe task, we thus add a sigmoid function on the predicted distance (where the sigmoid function is not centered on zero, but on a positive constant, as distances are always 1We have also tried to cast the probe as a regression directly pre2https://www.ncbi.nlm.nih.gov/mesh/ also contains its own categorization, composed of only 8 classes: Case report, Diagnosis, Forecasting, General, Mechanism, Prevention, Transmission and Treatment, with as for the MeSH a short description for each of them.
All our experiments are made on the LitCovid dataset, with 27,321 articles (Train-Val-Test split: 19,125 / 2,732 / 5,464) that have both LitCovid and MeSH annotations (several of the 8 classes from LitCovid + avg 13.5 MeSH/artors from Medline).
Training and evaluation relies on MeSH annotations (semantically richer), with results that reflects both fewshot for low frequency terms, and zero-shot results for 747 held-out MeSH descriptors. In a second step we also evaluate on LitCovid categories to test a transfer learning ticle out of around 9,000 COVID-19 related MeSH descrip- techniques to find “hard negatives”. In our case, the scenario where we change the categorization at test time. since we do not have scaling problem (only 8 labels).
create pairs ({, }, annotation.
We present in this section the adaptation of annotations for the “open input” architectures. The objective is to also add all the ancestors of the annotated MeSH terms as positive labels to overcome annotation incompleteness problem stated above. Adding the ancestors increases the size of the dataset by an important factor, so this is why this configuration is used for evaluation only.
The choice of the negatives is crucial in metric learning, and there have been lots of eforts given on developing S i b l i n g s configuration creates by its nature negatives that are dificult to distinguish from actual positives. Also we made the choice to use a binary classification layer, but losses like hinge loss or triplet loss could have been interesting in this particular case.
For LitCovid we consider all the {document, label} pairs ) from the original Zero-shot dataset creation. Inputs in zero-shot are diferent due to new class appearances: a document 1 being associated with two labels ( 1 and 2) will thus be transformed into two inputs ({ 1, 1}, positive) and ({ 2, 1}, positive). When a new label appears, it is enough to
is positive or not for the document 1. create the input ({
, 1}) to predict whether this label
To make the task meaningful, we also add negatives to both train and test datasets. However, in the case of MeSH classification, using all the negatives is not possible, for two reasons: (i) scalability problem: there are more than 9,000 labels and 27,321 documents, that results in hundreds of millions of combinations. (ii) data balancing problem: 9,000 negatives for 13 positives on avg. We therefore use following configurations: • B a l a n c e d : one random negative pair is added for each positive pair, to ensure a balanced distribution. So, given a document, we would always have the same number of positive and negative pairs. The negatives are sampled from all the possible negatives, based on the MeSH terms distributions to ensure that, given a MeSH term, we would also have the same number of positive and negative pairs. • S i b l i n g s : This configuration is only used in evaluation and aims at better disentangle errors due to “incompleteness of MeSH annotations” from real indexing errors. In this configuration, sib
The losses (binary cross entropy for the binary task and negative log-likelihood for the hierarchy generation task) are optimised using the AdamW and Adam algorithm with a learning rate of 2E-5 and 5E-4 respectively. Training is done over 4 epochs, with a save every 0.25 epochs. Best model is selected based on the validation loss. We used a batch size of 16, and performed 3 runs for each model (see standard deviation in the section 5).
The
BioBERT pretrained model we used was m o n o l o g g / b i o b e r t _ v 1 . 1 _ p u b m e d from Hugging Face. This model accepts an input sequence of up to 512 tokens, therefore extra tokens were truncated.
Concerning probing tasks, each MeSH-to-MeSH pair requires a gold distance (see section 3.2). For the Shortest-Path Probe they are computed using the Floyd-Warshall algorithm, while for the CommonAncestors Probe, they were deducted from Tree Numbers. Optimizer of the probe task is AdamW, with a 2.5E-5 learning rate. We also only focused on the “N”, “E”, “C”, “D” and “G” branches of the MeSH hiearchy, corresponding resp. to “Health Care Category”, “Analytical, Diagnostic and Therapeutic Techniques and Equipment Category”, “Diseases Category”, “Chemicals and Drugs Category” and “Phenomena and Processes Category” (they are the most representative of the dataset). From all possible MeSH-to-MeSH pairs, we have randomly selected 10% of them to reduce computation time. Validation and evaluation are performed on 30% of the MeSH descriptors, that we held out from probe training.
We present in this section the results in zero-shot and lings according to MeSH hierarchy of the pos- few-shot on both MeSH descriptors and LitCovid labels itive pairs are added with negative labels. We
from our two models BioBERT STL and MTL. We then discuss the results of the probing tasks, the architectures, the quality of annotations and also how we approached the problem of large scale zero-shot classification.
Medline/MeSH. The models have been trained on the b a l a n c e d configuration, and then tested on b a l a n c e d and s i b l i n g s . Table 1 compares results on those diferent test set configurations both in non zero-shot and zeroshot settings. Note, that as highlighted in section 4.1, the s i b l i n g s configuration is more dificult than the b a l a n c e d one, which explains a high gap between the F1-score from the two configurations. This is mainly due to a lower precision as the model tends to wrongly predict the siblings from the positive MeSH terms as positives examples (while we consider them as negatives in s i b l i n g s settings). On the b a l a n c e d test set, BioBERT STL has Figure 4: F1-score depending on the number of occurrences of a better F1-score both in zero-shot and non zero-shot tthraeinMinegSHdattearsmetinisthbaeltarnacineds.et. Evaluation dataset is siblings, settings, while on the s i b l i n g s one, the BioBERT MTL model performs better. This diference is due to the high precision of BioBERT MTL model in both settings. More precisely, the BioBERT MTL model seems to be better 5 shows F1-scores with respect to the deepness of the on dificult pairs, like in the s i b l i n g s settings where you MeSH descriptors both for BioBERT STL and MTL modmay have very close negative pairs (for example Breast els. As shown in the figure, F1-score tends to decrease Cyst positive and Bronchogenic Cyst negative). for more general terms (first 4 levels of hierarchy), but
Figure 4 plots the F1-score with respect to the number increases for more specific terms (after 4th level of hierarof occurrences of the MeSH descriptors in the train set chy). We believe this could be due to the incompleteness thus allowing to evaluate few-shot learning quality. First of annotations used during training of the MeSH descripwe note a clear increase of F1-score with the number tors. Recall that training is performed in the b a l a n c e d of occurrences of the MeSH terms in the dataset (up to setting, and therefore ancestor MeSH descriptors are not 0.7) on both models. This indicates, as one would expect, always explicitly annotated, which could result in low that the models are really struggling with dificult pairs performance on high hierarchy levels. This graph is also that contains rare MeSH descriptors. In comparison with dificult to interpret since some branches from the MeSH Table 1, the F1-score in zero-shot on the balanced pairs hierarchy are deeper than others, therefore “specificity” is of 0.76, while the F1-score for the rare MeSH terms is of term with respect to its absolute depth may be difermuch lower. We also note, that the BioBERT MTL model ent; the only information on depth is that deeper MeSH allows to slightly improve performance in low resources terms are in general more specific one. settings (for the terms occurring less then 10 times in the training data: 1 and (1, 10] bins in the figure). LitCovid. Table 2 reports results on LitCovid dataset Concerning the MeSH descriptors themselves, Figure in zero-shot setting for the representations obtained with Baseline IsIn Baseline Cos Sim BioBERT STL BioBERT MTL
F1-score are based on the mean error on the predicted distance with respect to the gold distance (shortest path length between two MeSH terms), while for the CommonAncestors Probe we report F1-score on binary tasks for each (evaluating whether the two MeSH have at least common ancestors).
tive models as representation of diferent MeSH in our experiments. We have compared this representation to both average pooling over the MeSH descriptors tokens and max pooling in our preliminary experiments, and STL and MTL models. On LitCovid, the STL model is bet- observed that C L S token was leading to the best perforter. We believe this may be due to LitCovid categories be- mance on the probe tasks. ing very general in comparison to the MeSH descriptors, As a baseline, we give in table 3 two other representatherefore the MTL model could not take advantage of its tions: BioBERT vanilla and Random. In BioBERT vanilla, better precision on more specialised pairs. In addition, representations are an average pooling of the MeSH outas previously, this could be due to the incompleteness put tokens provided by the BioBERT pretrained model of MeSH annotations, where only most specific MeSH without any finetuning (avg pooling was in this only case terms are present in training data, while LitCovid relies better than the C L S token3), and for Random, MeSH repon more generic labels. resentations are random representations sampled from a
We report two simple baselines for LitCovid dataset normal distribution. in Table 2: • Baseline IsIn where an abstract is associated with Comparison STL/MTL. Table 3 indicates that both a label that appears in the abstract itself (both STL and MTL model encode hierarchical structure of lower-cased); MeSH terms better than random baseline, but also bet• Baseline Cos Sim, where we take the C L S token ter that BioBERT vanilla baseline. More specifically, the representations of all labels (through a vanilla Common-Ancestors Probe implies that between two BioBERT), same for all abstracts, and then com- MeSH descriptors from the same categories, we have enpute the cosine similarity between each pair, with coded a common base, and there is a projection where a threshold defined on the validation set. MeSH descriptors from the same categories are closer to each others. Concerning the Shortest-Path Probe, reWe note that both BioBERT-based models perform signif- sults shows that there is also a projection where distances icantly better compared to those naive baselines, which (as shortest paths in the hierarchy graph) are respected. indicates that the models are able to exploit the semantics From the results, we also see that the additional task, of the label to some extent, and goes beyond simple label MTL model with the decoder block, is able to encode lookup in the abstract. We also see that the increase of even more hierarchical information in the C L S token, and F1-score is mainly due to a better recall which implies may be a hit to the better precision in zero-shot and better coverage of our models. few-shot results. Also, it is interesting to see that in the BioBERT vanilla model, there is already some good knowl
Finally, Table 3 reports the results of probing the learnt representations. Results for the Shortest-Path Probe 3possibly because in BioBERT vanilla model CLS token representations has not been finetuned for any task as opposed to our learnt models.
BioBERT vanilla Random BioBERT STL BioBERT MTL
BioBERT vanilla BioBERT STL BioBERT MTL Distance Error (std) edge about this hierarchical structure, which makes sense as this hierarchy is constructed on the semantics of the biomedical terms. 5.3. Limitations and possible future directions as we need to create too many pairs for a given document. Our approach was to work on a balanced subset of the possible pairs or a coherent one (resp. balanced and siblings configurations) for training and evaluation, however, this technique does not adapt to real world applications.
An interesting future direction could be combining our “open input” architecture with the high-coverage retrieval-like step which would first pre-select a subset of possible MeSH terms, and therefore restrict the search space for the second “open input” classification step. For example of the first step, a ColBERT model could compute representations of abstracts and classes independently instead of creating representations of pairs, and so reduce computational cost. Another possibility could be to use simple algorithms like BM25.
Other techniques in metric learning also exist, like triplet loss learning or hinge loss. Using triplets with “hard negatives” may help to learn a better representation space.
Multi-Tasks-Learning. When dealing with MTL, the In this work, we try to address the problem of zero-shot main dificulty comes from convergence speed of the classification that we defined as an open-input problem. diferent losses. In our framework, the main loss (clas- We compare a simple BioBERT model with a multi-tasks sification loss) converges faster than the secondary loss learning architecture that includes hierarchical seman(decoder loss), so we are not able to take full advantage tic knowledge. In zero-shot and few-shot settings, the of the decoder architecture. In a perfect scenario, the multi-tasks framework does not increase performances main task should be the harder one, but here it was not significantly. Still, we observe good results on precision the case, so we were forced to stop training earlier even and on structural probing tasks, which implies that the when using diferent coeficients and learning rates for addition of the seq2seq task has some beneficial efect in the two losses. Another possible future direction to ex- the ability of trained models to capture semantics. In parplore is to start training the decoder block before training ticular, the model is able to build a representation space the classification layer. where MeSH descriptors that have common ancestors are closer to each other, and where the overall hierarchical organisation of the MeSH is respected. It would be interesting to further investigate additional tasks to take even better advantage of hierarchical knowledge encoded in medical terminologies, and thus improve quality and robustness of models representations.
Annotations. When dealing with transfer learning across diferent datasets, the question of the quality of the annotation needs to be taken into account. Diferent annotation systems (even when documents are manually annotated) may have labels that have diferent coverage, and overlapping, which adds some bias in results. As an example, when we train our model on the Medline annotations and then test in zero-shot on LitCovid labels, results are dificult to interpret, because the scale and the coverage is completely diferent. [ 21] have studied the semantic interoperability of diferent biomedical annotation tools across multiple countries and databases, and they show that this was a real issue, that needs to be considered when dealing with such terminologies.
Large scale Zero-shot. “Open input” architectures are not adapted to very large scale zero-shot problems, ciation for Computing Machinery, New York, NY, ings of the 2019 Conference of the North American USA, 2020, p. 175–184. URL: https://doi.org/10.1145/ Chapter of the Association for Computational Lin3411408.3411414. doi:1 0 . 1 1 4 5 / 3 4 1 1 4 0 8 . 3 4 1 1 4 1 4 . guistics: Human Language Technologies, Volume [4] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. 1 (Long and Short Papers), Association for CompuSo, J. Kang, Biobert: a pre-trained biomedi- tational Linguistics, Minneapolis, Minnesota, 2019, cal language representation model for biomed- pp. 4129–4138. URL: https://aclanthology.org/N19ical text mining, Bioinformatics (2019). URL: 1419. doi:1 0 . 1 8 6 5 3 / v 1 / N 1 9 - 1 4 1 9 . http://dx.doi.org/10.1093/bioinformatics/btz682. [17] T. Gong, T. Lee, C. Stephenson, V. Renduchindoi:1 0 . 1 0 9 3 / b i o i n f o r m a t i c s / b t z 6 8 2 . tala, S. Padhy, A. Ndirango, G. Keskin, O. H. Eli[5] Q. Jin, B. Dhingra, W. W. Cohen, X. Lu, Prob- bol, A comparison of loss weighting strategies ing biomedical embeddings from language models, for multi task learning in deep neural networks, NAACL HLT 2019 (2019) 82. IEEE Access 7 (2019) 141627–141632. doi:1 0 . 1 1 0 9 / [6] W. Wang, V. W. Zheng, H. Yu, C. Miao, A survey A C C E S S . 2 0 1 9 . 2 9 4 3 6 0 4 .
of zero-shot learning: Settings, methods, and appli- [18] A. Kendall, Y. Gal, R. Cipolla, Multi-task learning uscations, ACM Transactions on Intelligent Systems ing uncertainty to weigh losses for scene geometry and Technology (TIST) 10 (2019) 1–37. and semantics, 2018. a r X i v : 1 7 0 5 . 0 7 1 1 5 . [7] J. Chen, Y. Geng, Z. Chen, I. Horrocks, J. Z. [19] Q. Chen, A. Allot, Z. Lu, Keep up with the latest Pan, H. Chen, Knowledge-aware zero-shot learn- coronavirus research, Nature 579 (2020) 193. URL: ing: Survey and perspective, arXiv preprint https://www.ncbi.nlm.nih.gov/pubmed/32157233. arXiv:2103.00070 (2021). doi:1 0 . 1 0 3 8 / d 4 1 5 8 6 - 0 2 0 - 0 0 6 9 4 - 1 . [8] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: [20] Q. Chen, A. Allot, Z. Lu, Litcovid: an open database Pre-training of deep bidirectional transformers for of covid-19 literature, Nucleic Acids Research language understanding, 2019. a r X i v : 1 8 1 0 . 0 4 8 0 5 . (2020). [9] I. Chalkidis, M. Fergadiotis, S. Kotitsas, P. Malaka- [21] J. A. Miñarro-Giménez, R. Cornet, M. Jaulent, siotis, N. Aletras, I. Androutsopoulos, An empir- H. Dewenter, S. Thun, K. R. Gøeg, D. Karlsson, ical study on large-scale multi-label text classifi- S. Schulz, Quantitative analysis of manual annocation including few and zero-shot labels, 2020. tation of clinical text samples, International Joura r X i v : 2 0 1 0 . 0 1 6 5 3 . nal of Medical Informatics 123 (2019) 37–48. URL: [10] A. Rios, R. Kavuluru, Few-shot and zero-shot multi- https://www.sciencedirect.com/science/article/pii/ label learning for structured label spaces, in: Pro- S1386505618305446. doi:h t t p s : / / d o i . o r g / 1 0 . 1 0 1 6 / ceedings of the 2018 Conference on Empirical Meth- j . i j m e d i n f . 2 0 1 8 . 1 2 . 0 1 1 . ods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium, 2018, pp. 3132–3142. URL: https://aclanthology.org/
D18-1352. doi:1 0 . 1 8 6 5 3 / v 1 / D 1 8 - 1 3 5 2 . [11] J. Wohlwend, E. R. Elenberg, S. Altschul, S. Henry,
T. Lei, Metric learning for dynamic text classification, 2019. a r X i v : 1 9 1 1 . 0 1 0 2 6 . [12] J. Risch, S. Garda, R. Krestel, Hierarchical document classification as a sequence generation task, 2020.
doi:1 0 . 1 1 4 5 / 3 3 8 3 5 8 3 . 3 3 9 8 5 3 8 . [13] A. Conneau, G. Kruszewski, G. Lample, L. Barrault,
M. Baroni, What you can cram into a single vector: Probing sentence embeddings for linguistic properties, 2018. a r X i v : 1 8 0 5 . 0 1 0 7 0 . [14] I. Tenney, P. Xia, B. Chen, A. Wang, A. Poliak, R. T.
McCoy, N. Kim, B. V. Durme, S. R. Bowman, D. Das, E. Pavlick, What do you learn from context? probing for sentence structure in contextualized word representations, 2019. a r X i v : 1 9 0 5 . 0 6 3 1 6 . [15] I. Alghanmi, L. Espinosa-Anke, S. Schockaert, Probing pre-trained language models for disease knowledge, 2021. a r X i v : 2 1 0 6 . 0 7 2 8 5 . [16] J. Hewitt, C. D. Manning, A structural probe for ifnding syntax in word representations, in: Proceed