Pre-trained language models have proven to be efective in multi-class text classification. Our goal is to study and improve this approach for multi-label text classification, a task that has been surprisingly little explored in the last few years despite its many real world applications. In this paper, our originality is to propose architectures for the classification layers that are used on top of transformers to improve their performance for multi-label classification. Our contribution involves the evaluation of thresholding methods on several transformers, either by computing an individual threshold for each label (IT ) or a global one (GCT ). We also propose two approaches for multi-label text classification. The first consists in adding a parameter for learning the number of labels present for a given example (NHA). The second approach consists in adding a layer to the classicfiation layers in order to learn the features for selecting the relevant labels while avoiding the use of thresholds (TL). We evaluate these approaches on two English corpora of newspaper articles and scientific papers and then on a new multi-label dataset of French scientific article abstracts publicly available. The evaluations show that the performance of our proposals exceeds that of state-of-the-art multi-label text classification methods for the evaluated datasets, and are transposable to any multi-label classification problem.
Multi-label classification is a generalization of the multi-class classification problem, where each instance is associated with exactly one label. In multi-label text classification, the goal is to associate one or more labels to the input text sample. It is an important natural language processing task that has many applications in other NLP tasks such as question answering or entity recognition, but also real-world applications such as information retrieval (e.g. metadata enrichment and analysis in digital libraries) or content recommendation, and so on.
Multi-label text classification is a challenging task due to the fact that several factors must be taken into account such as the dependencies that can exist between the labels, the complexity of extracting semantic features from the noisy input text that can contain redundant information, and mapping those features to multiple targets, while also finding the discriminative information that allows the identification of each label of the document.
Several methods have been proposed to tackle the multi-label classification problem, whether
it’s traditional methods such as Binary Relevance [
Few multi-label text datasets are popular among the papers treating the multi-label problem.
AAPD [
In this article, our main contributions are : • The creation of a French corpus of multi-label text classification, MFHAD (for Multilabel French HAL Abstracts Dataset), containing the abstracts of scientific articles obtained from the open archive HAL (https://hal.archives-ouvertes.fr) that contains more than one million papers, • The adaptation of available transformer models for multi-label classification, • The study of threshold selection methods for more eficient exploitation of the results of the transformer models, in particular, the choice of a global threshold , or a threshold specific to each label for the individual optimization of the labels, • The proposal of two alternative approaches to thresholding for the selection of relevant labels. The first one consists in introducing a parameter at the last layer of the transformer which will be trained for the calculation of the number of labels present in an example, the value of this parameter will be used to select the labels having the strongest activations; The second one consists in adding a final layer to the model, which will have the same number of parameters as the second last layer (equal to the number of labels), in order to obtain more discriminating activation values for the given example, i.e. high activation if labels are present, low activation in the opposite case.
The article is organized as follows: the section 2 presents the approaches that address the multi-label classification problem, the sections 3 and 4 describe the proposed approaches and the section 5 is dedicated to experiments.
In multi-class classification, each example (instance) in the dataset is associated with a single label. Multi-label classification is furthermore about being able to associate each entry with multiple Y labels, rather than just one.
Multi-label classification methods can be classified into three categories: problem transformation, adaptation, and ensemble methods.
Problem transformation consists in ’transforming’ the dataset to change the problem into
a single-label multi-class classification. One such method is to consider all possible unique
combinations of labels, label powerset [
A set of multi-class classifiers can be combined to create a multi-label classifier. For a given
instance, each classifier will predict a single label and all outputs of these classifiers are then
combined via an ensemble method. One of these methods consists in considering a label as
present if a percentage of classifiers having predicted this label is reached, also called the
discriminative threshold. The algorithm [
Problem adaptation methods do not require a transformation of the dataset but an adaptation of
classification algorithms, such as ML-kNN [
The adaptation of deep learning algorithms for multi-labels remains in general an avenue with few contributions. An adaptation of these approaches could contribute to a significant increase in performance. The use of a single model without the need for prior data transformation is an eficient method to try to address the multi-label problem.
Thresholding methods directly impact the choice of a label for the multi-label problem. The threshold can be adjusted in several ways, either to optimize all the labels (a global threshold), or to optimize each label individually (number of thresholds equal to the number of labels). Let be the number of examples in the test dataset (or validation) and the number of labels. The four most commonly used strategies for choosing the threshold(s) are:
- SCut: Labels are optimized individually, thresholds are chosen based on the validation set,
measured either by maximizing a score or minimizing a cost function and without guaranteeing
a global optimum. This method can also be used to obtain a global threshold;
- RCut (Rank Cut) : Labels are ordered according to their score, the first labels are chosen
as relevant labels. The parameter is either predefined or set from the validation dataset [
Variations of these methods aiming to overcome the constraints they may impose have been
proposed [
…
…
C (1) C (0) = [CLS] pooler
…
C (L) Transformer Layer-12 Transformer Layer-2 Transformer Layer-1 ny activations : al(L)
Transformer embeddings
[
On the other hand, MAGNET [
Transformers have been used for the ’extreme’ multi-label classification of text where very large corpora of texts with a number of labels that can reach tens of thousands are processed. Such architectures are not well suited for short texts.
Attempts to use neural networks for text classification do not focus on multi-label classification. Those that do deal with this problem generally do not give importance to how the output layer activations are exploited (e.g. use of thresholds).
BERT introduces bi-directionality in the prediction of masked tokens, where both the left and right semantic contexts of the word to be predicted are considered. In addition to masked language modeling, BERT is trained for next-sentence prediction, a task where the model receives a pair of sentences and tries to predict whether the second sentence follows the first. BERT introduces a special classification token [CLS] (also having an identifier and an embedding vector) containing a hidden state of the sentence, updated in each layer of the model.
A feed-forward neural network (FFNN) of dense layers (usually =2) is added on top of the
last transformer layers of the model. This is done to fine-tune the pre-trained transformer for
the desired NLP task (text classification in our case). The token [CLS] is the input of this FFNN,
and outputs corresponding to the labels of the dataset (similar approach to [
For multi-label classification, the values of the [] activations of the [] output layer can be used to determine the presence of a label. Each activation can be a value between 0 and 1 representing a probability of the presence of the corresponding label. A threshold is then used for the label selection process, the trivial value of 0.5 is usually used for this purpose. The Sigmoid function is the activation function that is suited for this case, alongside the Binary Cross Entropy as a loss function.
This approach cannot be considered completely as a problem transformation, it does not require a transformation of the dataset or the creation of multiple binary classifiers. It also does not a complete adaptation of neural networks for multi-label classification, the results must be processed later on for the final classifications.
#of Labels
Parameter … …
… activatinoyns :al(n) C (0) ([CLS]) C (1)
C (n) Transformer
MAE / MSE BCE … … …
… C (0) ([CLS]) C (1)
C (2) C (n) ny activations ny activations
Thresholding approaches can be applied to the transformer if we consider each activation in the ifnal layer to be a representation of a binary classifier of the label it represents. If the activation is high, the label is considered to be a valid label, and vice-versa.
We propose two alternative methods to thresholding in an attempt for more eficient use of the [] activation values of the output layer. These approaches aim to avoid thresholding by learning text-specific features for a better label selection.
The classification threshold can be chosen to maximize the classification scores. During the training of the model and after each iteration, the value of is varied from 0 to 1, with a step defined beforehand ( 10− 2). The micro-F1 score is then computed for each threshold . We ifnally obtain the global optimal threshold which provides the best performance on the training dataset. This threshold is then used for the validation and test datasets. The set of labels present for an example can be expressed as: ∈ = ∪∈ {} : ([])) ≥ (1) [] being the activation corresponding to label among the activations []. A variant of the SCut method (with a global threshold) consists in fine-tuning the optimal threshold from the validation dataset and then applying it to the test dataset. Only the first approach has been studied in this article.
The use of a shared threshold for all labels assumes that the features associated with the activations [] are the same for each label, which is not the case, the activation intensity of a [], when the label is present, varies from label to label, an efect neuron in the last layer that is accentuated if the dataset is unbalanced.
We, therefore, propose to evaluate the SCut method (with individual thresholds) by setting a threshold for each label in the dataset, assigned to the [] activations of []. The values of are the values that maximize the classification scores for a label . These thresholds are computed as for during the training phase of the model, and on the training dataset, by varying each threshold to maximize the F1 score for each label.
∈ = ∪∈ {} : ([]) ≥ (2)
Using a threshold to determine the presence of a label leads in several cases to an underclassification (respectively over-classification) of an instance when the predicted number of labels is lower (respectively higher) than the actual number of labels. The first proposed alternative consists in introducing a neuron at the last layer which will be used only for computing the number of labels present for an instance. Let ′[] be the list of the highest activations, and being the actual number of labels present for a given instance: ∈ = ∪∈ {} : ∈ ′[]) []) (3) The number of labels present for an instance is the target value, the mean absolute error MAE is used as a loss function for this regression problem. We use a single optimizer in the backpropagation, therefore, the error of the regression must be scaled to match the classification error, done by reducing the regression error by a factor of 5. The value of this neuron will be used to recover the highest activations that will be considered as predicted labels, similarly to the RCut method. Figure 2a shows the model architecture for this approach.
The objective of this architecture is to extract from the [CLS] token, criteria or features about the number of distinct topics present in the text. This token, the final output of the transformer layers, contains an information-rich representation of the input text.
The second proposed approach is based on the addition of a dense layer, after the output of the classifier, for a total of = 3 layers, having neurons to match the last classification layer of the model. This additional layer aims to make the values of the final activations as close as possible to 1 in the case of the presence of the label, or to 0 in the opposite case. Therefore, the trivial threshold of 0.5 can be used for the classification.
The addition of this layer could result in an increase in the recall of the classifier as many activations of the non-present labels will no longer exceed the classification threshold since they will be as close as possible to 0. A gain in precision can also be expected as the activations of the present labels will be emphasized more than the activations of the irrelevant labels.
For this approach, we use two optimizers, the first one which includes and optimizes all the layers of the transformers as well as the first two layers of the classifier, and another one dedicated to the optimization of the last layer that we added. The error function used is the same for both parts of the classifier, the BCE in this case. The final architecture of the classifier is presented in the figure 2b
We present in this section the results of the evaluation of non-deep learning baselines as well as all the previously mentioned methods on three multi-label text datasets, including the French dataset "MFHAD" that we have designed and made available. We compare the diferent methods, i.e. the thresholding methods as well as the proposed alternatives coupled with BERT transformer and its variants, to baseline approaches, all put in perspective with optimal target results (oracle approaches).
For the evaluation of the proposed methods, we use HugginFace’s [
- RoBERTa [
- DistilBERT [
The comparison of diferent transformers is important to evaluate the performance of our approaches as well as their re-usability and applicability on diferent transformer architectures.
Multi-label text classification is not present in NLP leaderboards, the GLUE benchmark for example, nor in recent CLEF or Semeval conferences. There are few multi-label corpora frequently used in the literature that can be used to evaluate and compare models, and this is even more true for the French language. To remedy this, we have built our own corpus comprised of abstracts of scientific articles written in French from HAL, a corpus that we make available to the public. In this section, we provide details about the diferent datasets used 1 for the evaluation of the diferent models :
- MFHAD2: Our dataset is comprised of abstracts of french scientific papers collected from ’HAL’, an open academic research archive, distributed over three major scientific domains, i.e. computer science, physics, and mathematics. Articles published between 1980 and 2021 that have a french abstract, which represents 15 771 documents distributed over 200 diferent labels; - Reuter-21578 3 is a collection of articles from the Reuters newswire from the year 1987. It is a dataset that has been often used to evaluate models for multi-label text classification. An article can belong to one or more of the 90 domains of the dataset;
- AAPD (or ArXiv Academic Paper Dataset) is, not unlike the "MFHAD", a collection of the
"Abstract" of several scientific publications. An article can have one or more classifications
among 54 labels. We use the same training, validation, and test distribution as [
Table 1) and figure 3 present the diferent characteristics of these datasets in more detail.
We will compare the two proposed approaches as well as the coupling of thresholding methods to transformers with methods that are more explicit on the label selection criteria, but also with other adaptations of deep learning models to multi-label text classification. We will also compare all these methods to several upper bounds that represent the optimal results that can be achieved. 1All datasets can be downloaded here: https://zenodo.org/record/6344750#.Yio1YH_MK-r 2The extraction of these abstracts was made using the tool made available by HAL: https://api.archives-ouvertes.fr/ docs/search 3https://archive.ics.uci.edu/ml/datasets/reuters-21578+text+categorization+collection
We will first do a comparison with non-neuronal approaches, with interpretable machine learning classification criteria, using TF-IDF extracted features (with no maximum number of features for each dataset), as inputs:
- Decision trees using the "Gini" criterion, without a maximum tree depth, and 2 as the minimum samples for splitting nodes;
- Random Forest with 100 estimators (number of trees) and the same parameters used in the previous method for each tree; - Bagging using decision trees as main estimator (10 estimators); - GradientBoosting with logistic regression as error function, a learning rate of 0.1 and 100 estimators; - Support Vector machine using RBF as the kernel and a regularization parameter of 1.0.
We also include deep learning approaches as a comparison for the evaluation of our approaches:
- CNN [30] and CNN-RNN [
- SGM [
- MAGNET [
- DocBERT [
To evaluate the performance gain of our approaches, we compare them to an unchanged version of each transformer model, where no architecture modification is applied, and the classification threshold is the trivial value of 0.5.
The target theoretical optimums (oracle approaches) are represented by the following two approaches:
- The oracle approach for the N highest activation, where we consider the number of labels present for an instance as a given and then take the highest activations as the present labels; - The oracle approach for both SCut thresholding methods, where the calculation of the global threshold and the individual thresholds is done from the test dataset. Considered as the optimal results towards which these methods should come as close as possible.
In this section, we present the performances of all the approaches mentioned in the previous sections, tested on the diferent transformers presented in section 3. We propose the following notations for these approaches: - "GCT": The Global Optimal Threshold method (see section 4.1); - "IT": The Individual Threshold method (see section 4.2); - "NHA": The N largest activations approach (see section 4.3); - "TL": Denotes the "thresholding layer" approach (see section 4.4). As for the oracle approaches, they will be designated by adding after the corresponding approaches.
The maximum length of the sequences used is 512 tokens for all datasets. Tables 2 and 4 show the micro-F1 (with precision and recall) and Accuracy scores for the English and French corpus test datasets respectively. The tables 3 and 5 present the optimum scores for the diferent approaches. The results of the thresholding methods as well as the proposed architectures were obtained from the base versions of the transformers used.
Transformers outperform all other methods, whether it is the classical methods, or other deep learning methods. Our baseline versions of BERT and its variants achieves a better performance than the base version of DocBERT (the current state of the art in multi-label text classification for the AAPD and Reuters corpora), and for Reuters, better than its large version. This is due to the longer training in the fine-tuning process, 150 epochs for the Reuters dataset vs 30 in the case of DocBert, and 40 epochs for the AAPD dataset vs 20. For transformers, a longer training process generally yields better performance with a low risk of over-fitting.
We also note that SVMs and decision trees obtain the highest micro-precision scores, at the expense of recall rate, thus lowering the micro F1 score. But precision is not a reliable factor for performance measurement in multi-label classification. SVM can be considered as the best performing non-neural approach due to its high accuracy score, but it falls short of the other methods tested.
The GCT thresholding techniques seems to be the best among the evaluated approaches, it manages to get a better micro-F1 score than its thresholding counterpart, namely the IT approach, with a score of 91.02 vs 90.70 for the Reuters dataset, and 74.08 vs 74.04 for AAPD (see table 2). For the Reuters Dataset, the thresholding techniques and the proposed alternatives do not manage to achieve a gain in micro-F1 scores compared to the baseline versions, except for the RoBERTa model. This does not apply to the AAPD dataset, gains in performance can be perceived in using thresholding techniques or the TL approach. Especially for the RoBERTa model where the gain is the highest. The large version of DocBERT remains the best performing
Thresholding without any specific architecture (GCT and IT)
Transformer Architecture Adaptation (NHA et TL) Models model for AAPD. The complex nature of the scientific vocabulary of this corpus underlines the possible benefits of increasing the size of the transforming models (adding several layers and increasing the embedding dimension).
The diference in performance gains between the datasets can be due to their complexity level. The proportion of shared tokens/words between the dictionary of the transformers and the Reuters corpus is much higher than AAPD due to its scientifical nature, which makes the training process much more dificult on the latter, therefore rendering the thresholding and alternative approaches more useful to gain performance on the multi-label classification task. Those techniques won’t be as useful if the learning process yields a good overall semantic understanding of the dataset, in that case, a threshold of 0.5 is generally more than enough.
The RoBERTa model is the lowest performing variant of BERT. But the performance gain on RoBERTa is more perceivable for all the evaluated datasets. This can be explained by the fact that for this variant of BERT, the Next-Sentence Prediction objective is removed in the pre-training process. The goal of NSP is to learn long dependencies that can exist across sentences, whereas Masked Language Modeling is more focused on understanding relationships on the word level. For multi-label text classification, distinguishing the diference between domains across multiple sentences is vital, and NSP is certainly a good contributing factor for this purpose.
The same conclusions can be drawn from the performance data on the MFHAD dataset, where the baseline version of FlauBERT outperforms its CamemBERT counterpart (65.92 vs 64.47 in the F1-score). This comes as no surprise knowing that the latter is a variant based on RoBERTa that also lacks the NSP objective, but also the fact that FlauBERT is pre-trained, in addition to text crawled from the internet (Common-Crawl), on wikipedia article and books, thus getting more vocabulary coverage of MFHAD than CamemBERT that is pre-trained on a corpus derived from Common-Crawl. The evaluated approaches fail in obtaining a gain in performance for FlauBERT that achieved good performance with its baseline version but allow CamemBERT to get closer to FlauBERT ’s performance.
The TL architecture performs better than the base version of the DocBERT and MAGNET models, and matches the large version of the former with a score of 90.70 for the DeBERTa model on the Reuters dataset. The NHA approach does not succeed in reaching its theoretical optimal, even though it manages to obtain a score exceeding the base version of DocBERT for the same dataset (see Transformer Architecture Adaptation part of table 2). These two architectures require less training time than the two thresholding methods (1.5× faster), but have a higher inference cost, especially for TL where it’s almost twice that of the other approaches, due to the presence of the extra layer that this architecture requires.
As shown in tables 2 and 4, an increase in the recall is observed for almost every model and every dataset for the two thresholding methods. This is due to the fact that modifying the classification thresholds (especially by lowering them) can lead to more labels being predicted as present, but can also lead to a decrease in precision. The same can be said for the TL alternative method where adding an additional layer contributed in some cases to an increase in the final activations’ values, thus using a threshold of 0.5 might include some labels that couldn’t otherwise be predicted. The opposite efect is observed for the NHA method, where the model predicts the most frequent number of labels present in the unbalanced dataset which is always the smallest. This leads to an increase in precision (fewer labels are being predicted thus fewer false positives) but this comes at the cost of the recall.
The IT approach is the best performing among the evaluated oracle approaches. Computing an individual threshold for each label from the test set leads to a global optimal score for all labels, surpassing the optimum for the GCT method. But this is not the case for its experimental equivalent. GCT remains the method which comes closest to its theoretical optimum (cf. tables 3 and 2). This can be explained by the fact that for the IT method several thresholds must be computed for each label, which does not guarantee a global optimal on all the labels for the test set. This efect is amplified if the number of labels is large.
For the theoretical optimal of the method NHA, providing the actual number of relevant labels allows new labels to be considered as present, which increases true positives and decreases false negatives. But this could lead to an increase in false positives as these new labels are in some cases non-valid predictions. The instances for which this approach reduces the number of what would have otherwise been predicted by the model (using 0.5 as a threshold) are rare. Thus the micro F1 score in some cases may be lower compared to the theoretical optimal of the thresholding approaches. On the other hand, a considerable increase of the accuracy is observed, it is in fact the oracle approach that achieves the highest accuracy scores as shown in tables 3 and 5. The actual implementation of this approach did not achieve the same performance as its theoretical optimum. The hidden state of the sentence contained in the [CLS] token may not be suficient to estimate the number of existing labels, a task that is made harder by the unbalanced nature of the datasets (regarding the number of labels per instance). Exploiting the attention scores obtained in the diferent layers of the model could be a more eficient method to accomplish this task.
DeBERTa is the best performing variant of BERT among all the evaluated models, the changes made by this variant regarding the addition of the relative position of the word to the sentence as an input, seem to improve the performance of BERT. This performance improvement comes at the expense of the training speed of the model (18% slower training speed than the other variants on average, but with similar inference speeds). As for DistilBERT, it achieves results at the same level as its original version, despite its reduced size. Knowledge distillation seems to be an eficient method to overcome the major drawback of transformers and neural networks: the necessity of a long training time.
As far as the application domain is concerned, multi-label classification is a relevant task. However, multi-label text classification is not a common task and, regrettably, it is not included in the most prominent benchmarks such as GLUE.
Transformer-based language models outperform other deep neural architectures and provide a strong adaptable foundation for a multitude of NLP tasks and text classification, which is the direction we followed in this study.
First, we have tested and shown in this paper that thresholding approaches can achieve a performance gain if the learning process does not yield good results for datasets of a complex nature. Computing a global threshold achieves higher results than computing an individual threshold for each class. The optimization done on each class does not guarantee a general optimal for all labels. We then proposed modifications to the transformer architecture. This is done by adding an additional layer, with a number of activations equal to the number of classes, at the bottom of the model to avoid optimizing thresholds. An approach that, on average, is as efective as thresholding approaches. The analysis of the optimal to be reached showed that the use of the number of classes for the selection of labels significantly increases the performance. However, the diference between the optimal and the actual results of our experiments shows that our proposal needs to be improved in the case of unbalanced data sets. The language of the text corpora, English for AAPD and Reuters, or French for the MFHAD corpus we have built, does not seem to be a factor that impacts the performance of the proposed approaches. These approaches can be used for any multilabel classification problem. Each BERT variant seeks to improve the constraining aspects of the original model. DistilBERT, despite its reduced size, obtains results as high as the original version. DeBERTa is the best performing variant, together with CamemBERT which outperforms FlauBERT.
We have shown the potential of approaches for adapting the transformer model for multi-label text classification. All these approaches are not limited to text classification but can be used for any other multi-label classification task. Our future work will be focused on exploring other methods of adapting neural networks for multi-label text classification. The thresholds can be learned parameters of the model during the training phase. Finding a more efective method for computing the number of classes present for the NHA method, and finding a better way to represent the dependencies between the labels for the TL approach, are other areas we will explore in the future. [28] P. J. O. Suárez, B. Sagot, L. Romary, Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures, Leibniz-Institut für Deutsche Sprache, 2019. [29] H. Le, L. Vial, J. Frej, V. Segonne, M. Coavoux, B. Lecouteux, A. Allauzen, B. Crabbe, L. Besacier, D. Schwab, FlauBERT: Unsupervised Language Model Pre-training for French, in: LREC, 2020. [30] Y. Kim, Convolutional Neural Networks for Sentence Classification, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Doha, Qatar, 2014, pp. 1746–1751.