Generative language models, particularly adopting text-to-text frameworks, have shown significant success in NLP tasks. While much research has focused on input representations via prompting techniques, less attention has been given to optimizing output representations. Previous studies found inconsistent efects of label representations on model performance in classification tasks using these models. In this work, we introduce a novel method for selecting well-performing label representations by leveraging the attention mechanisms of Transformer models. We used an Italian T5 model fine-tuned on a topic classification task, trained on posts extracted from online forums and categorized into 11 classes, to evaluate diferent label representation selection strategies. We've employed a context-mixing score called Value Zeroing to assess each token's impact to select possible representations from the training set. Our results include a detailed qualitative analysis to identify which label choices most significantly afect classification outcomes, suggesting that using our approach to select label representations can enhance performance.
In recent years, generative language models have become increasingly prevalent for solving a wide
range of NLP tasks. Among these models, the text-to-text paradigm has demonstrated significant success
across numerous applications [
Starting from these premises, in this work we propose a novel methodology for selecting label
representations in a text-to-text classification scenario exploiting the potential of the attention mechanism
of Transformer models. In fact, previous work showed that attention can be successfully employed
in several scenarios, such as in the automatic identification of keyphrases from documents [
To investigate this, we conducted our experiments by fine-tuning the IT5 model on the topic classiifcation task [ 15] using various label representations. Specifically, we tested diferent approaches for selecting candidate labels relying on Value Zeroing [16], a context-mixing score based on the attention mechanism aimed at quantifying the contribution each context token has in determining the final representation of a target token. Moreover, we performed a thorough qualitative analysis to determine which labels have the most substantial impact on the improvement or decline of classification results. Contributions In this paper we: i) present a novel technique for label representation selection based on the attention mechanics of Transformers models. We tested three diferent configurations and found that one shows promising results in finding the best possible representations to maximize performances; ii) show an in-depth qualitative analysis of the chosen representation, with the intent to find usable correlations to improve the performance of our label representation selection technique.
When employing a text-to-text model for classification tasks, the class names must be represented as specific sequences of tokens (hereafter label representations) that the model outputs to assign an input to a particular class. We aim to find a set of suitable label representations that maximize the model’s performance.
To do so, we hypothesize that we can use the attention mechanism of the model to find suitable representations for each class inside the training set of the target task. Particularly, we look at which tokens were the most salient for building the vectorial representations of important tokens in the post using Value Zeroing. We tested three diferent ways to select the important tokens inside the posts. First, we tried looking at the tokens that were used to build the representation of the End-of-Sentence special character of T5 </s> (EOS). Then we also tried to append class-related tokens to the end of the posts. The idea was to inject class-related words into the posts to see which original tokens from the posts were useful in building them: • In the Appended Label method, we define as the translation of the original class names1, e.g. the post “Che giornata indimenticabile... è passato proprio tanto tempo!</s>” from category SPORTS, becomes: “Che giornata indimenticabile... è passato proprio tanto tempo! Sport</s>” ; • In the Appended Label with Prompt method, we provide the model additional context, by defining as: La frase precedente appartiene alla categoria (English translation: The previous post belongs to the category of ) followed by the original class name translated, e.g. the post “Che giornata indimenticabile... è passato proprio tanto tempo!</s>” from category SPORTS, becomes: “Che giornata indimenticabile... è passato proprio tanto tempo! La frase precedente appartiene alla categoria Sport</s>”
Formally, let ∈ as one of the training posts in the dataset . Each post is tagged with one of the classes ∈ where is the set of the possible topics. The posts are tokenized using the provided IT5-trained tokenizer . For each post , we injected a series of tokens tokenized with . The 1List of translated labels: anime, automobilismo, bicicletta, sport, natura, metal detector, medicina, celebrità, fumo, intrattenimento and tecnologia. objective is to study which tokens from the original post are more salient for the model to build the representation of the tokens in .
As explained before, the diference between the three methods is how is defined: in the EOS method = </s>, in the Appended Label method is equal to the appended and translated class name, and in the Appended Label with Prompt method is equal to the predefined prompt completed with the translated class name.
After injecting in each ∈ we pass each post in inference through our modified implementation of IT5, whose Encoder is able to calculate the Value-Zeroing matrix (see Section 3.1). Then, we define as a candidate label representation the token ∈ that obtained the highest Value-Zeroing score with respect to the tokens in :
= (__(, )) By doing so we obtain, for each post, the most important token whose embedding vector is used to construct the representation of 2. After doing it for the whole dataset, we obtain, for each category ∈ , a list of representations . contains tuples equal to the number of posts in the dataset tagged with . Each tuple is composed by the candidate label representation and the Value-Zeroing score it obtained with respect to , = [(1, 1), ..., (, )].
Since some of these representations may be duplicates (i.e. the same representation has been chosen from multiple posts), we decided to aggregate those representations, in a way that rewards their higher frequency count. We aggregate all the tuples that have the same representation and sum together their Value-Zeroing score creating a single element in the list. After doing these aggregation steps for all categories ∀ : ∈ , we have, for each category , a set of representations that we sort based on the value of the tuples in descending order, obtaining a ranked Representation Set.
Finally, we define a set of representations , called the Representation Set of rank where, for each category , we have the ranked representation in . E.g. in the set 0, for each category, we have the best-ranked representation, while in the set 10, for each category we have the representation that ranked 11th.
An overview of our approach is illustrated in Figure 1. 2Since Transformers’ tokenizers split tokens in multiple subtokens, to obtain the full word, we reconstruct it by reconnecting all the tokens that are part of the word that the token with the highest Value-Zeroing score is from. The Value-Zeroing score we consider for the full word is the one of the token that was selected. We decided to avoid aggregating the score of the full word in any way, because that could reward or punish multi-token words .
We tested our approach to solving a topic classification task by training our models on forum posts categorized into 11 classes. We tested all three previously presented selection methods and evaluated their performances: we used as the target output the first ten best-performing Representation Sets 0, 1, ..., 9 for all three methods, training ten models for each, for a total of 30 trained models. Then, having assessed that the most promising strategy was the EOS method, we trained 100 models using the Representation Sets 0, ..., 99 extracted with the EOS method to study the efectiveness of this+ approach.
In the following sections, we detail how the Value Zeroing technique works (Sec. 3.1), we present the data, the model and the evaluation methods used in our experiments (Sec. 3.2 and 3.3).
Value Zeroing [16] draws inspiration from traditional interpretability techniques, where the influence of a feature (in this case, a token representation) on the model’s output is extracted by removing that feature from the input, i.e. feature importance methods [17]. Since deleting a word from a sentence, without changing the semantics of it, is either challenging or impossible, the method opts to eliminate it during the Attention computation of the considered layer, by zeroing its value vector, i.e. setting each element in the vector to 0. Inside the Self-Attention layer of a Transformer, for each Attention head ℎ, the input vector x, for the ℎ token in the sequence is transformed in three distinct vectors through the use of diferent sets of weight: the Query vector qℎ, the key vector kℎ and the Value vector vℎ. The context vector zℎ for the ℎ token of each Attention head is generated as a weighted sum over the Value vector:
zℎ = ∑︁ ℎ vℎ
=1 C = (x¬ , x) (1) (2) where ℎ is the raw Attention weight assigned to the ℎ token and computed as a Softmax-normalized dot product between the corresponding Query and Key vectors. In Value-zeroing Equation 1 is changed by replacing the Value vector associated to with a zero vector vℎ ← 0, ∀ℎ ∈ , where the context vector for the ℎ token is being computed. This provides a new representation x¬ that has excluded . By comparing the original representation x with this new one, usually by means of a pairwise distance metric, we obtain a measure of how much the output representation is afected by the exclusion of . In our experiment, we chose the cosine distance as a distance metric: Computing Equation 2 for each token , generates a Value-Zeroing Matrix C where the value of cell C in the map indicates the degree to which the ℎ token is dependent on the ℎ to form its contextualized vectorial representation.
For our experiments, we modified the implementation of T5 in the Python transformers library 3 such that the model’s encoder can calculate the Value-Zeroing Matrix C. In particular, we look at the section of the matrix C:,0: where is the number of tokens in the original sentence and is the number of tokens that compose the appended specially placed tokens (See 2 for how we chose these tokens). This section of C illustrates how each original token in the sentence contributes to the vectorial representation of the appended tokens. 3.2. Data We relied on posts extracted from TAG-IT [15], the profiling shared task presented at EVALITA 2020 [18]. The dataset, based on the corpus defined in [ 19], consists of more than 18,000 posts written in 3The modified class is T5ForConditionalGeneration available in https://github.com/huggingface/transformers/blob/main/src/ transformers/models/t5/modeling_t5.py. To do so, we adapted the original Value Zeroing implementation for the BERT transformer modelling class: https://github.com/hmohebbi/ValueZeroing.
Categories Anime Auto-Moto Bikes Celebrities Entertainment Medicine-Aesthetics Metal-Detecting Nature Smoke Sports Technology All
Italian and collected from diferent blogs. Each post is labeled with three diferent labels: age (binned into 5 classes) and gender (male or female) of the writer, and topic (11 classes).
Since previous works have shown that tasks that are solved through the use of lexical and semantic
information benefit the most from a well-chosen label representation [
We used the T5 base version pre-trained on the Italian language, i.e. IT54. In particular, the model was trained on the Italian sentences extracted from a cleaned version of the mC4 corpus [20], a multilingual version of the C4 corpus including 107 languages.
Models’ performances on the topic classification task were computed using the F-Score on the test set. To evaluate the capability of our selection method to find suitable labels, we trained up to 100 models with 100 diferent Representation Sets. Each of these sets was composed of representations chosen by our method and was ranked based on its prediction, from the set predicted as the best (Rank 0) to the set predicted to be the worst (Rank 99). We then calculated the Spearman correlation between the set’s ranking, and the obtained F-Score using that set. If our method can reliably predict the best representation to maximize performance, we expect a correlation between the ranking and the model performances. We used the traditional approach of using translated class names for classification as our baseline.
As a first step, we evaluated the first ten Representation Sets 0, ..., 9 from each of the three tested methods to assess their potential in predicting the most efective representations. Figure 2 reports the scatter-plots showing the F-scores obtained on the test set by each model according to the 10 Representations Sets. As we can notice, the first two methods, Appended Label and Appended Label with Prompt, don’t show any particularly interesting trends. The first one has a slightly negative coeficient and a Spearman Correlation of 0.03 with a p-value of 0.934. With such a low correlation value and high p-value we can’t reject the null hypothesis and the obtained trend is probably random. The same can be said for the second method too, where we have a slightly positive trend, with a Spearman correlation of 0.151 with a p-value of 0.67. On the contrary, the third method shows a more pronounced negative trend ( = − 0.552), i.e. as the rank increases the performance of the models tends to decrease. Although the p-value of the correlation is below the standard cut-of threshold ( − = 0.098), we decided to use the EOS method for testing with a total of 100 representation sets. Before proceeding, we removed from the original dataset the posts belonging to TECHNOLOGY. This was done since for this class we extracted only 23 sets, due to the small number of samples in the training set. After removing this class, we evaluated the method with the rest of the categories training 100 models with the first 100 ranked Representations Sets.
Correlation results are reported in Figure 3, while Table 2 shows the performances of the models obtained with the representation set of rank 0, the best performing model (Ranked 20th), and the worst performing model (Ranked 95th) along with the baseline (A IT5 trained with the original class label translated into Italian). As we can see, the negative trend between models’ performance and Representation Sets observed previously can still be noted, although less pronounced ( = − 0.314, − = 0.001). In terms of classification scores, we obtained a diference of 0.05 in terms of F-score between the best-performing model obtained by rank 20 (0.68), and the worst-performing one obtained by rank 95 (0.63). Although general conclusions about the method cannot be drawn, it appears that, in this setting, selecting labels from the training set using Attention attribution techniques, such as Value-Zeroing, efectively identifies keywords with meaningful semantic connections that IT5 can leverage to achieve higher performance.
Interestingly, the lowest-performing model using the EOS method achieved the same F-score (0.63) as the baseline method, i.e. the standard approach of using translated class names. From this perspective, the EOS method demonstrates superiority over the standard approach: in fact, the model trained on the Representation Set ranked 0, identified by the EOS method as the best set, achieved an F-score of 0.656. While this is not the highest score produced by the EOS method, it still outperforms the standard approach. A possible explanation of the efectiveness of the EOS method could be that for building the </s> character, the Encoder of the model uses particularly informative words that we can leverage if used as label representation. The role of the EOS character and other similar characters that are used for modeling purposes, like the [CLS] character in BERT-like models, is to be used as input for the final Language Modeling classifier. That pushes the model during the pre-training phase to learn to construct a representation of such a token that summarizes all the relevant information in the sentence that is needed to complete the language modeling task [21]. So, by taking the highest Value-Zeroing score for constructing </s>, we find tokens that are usually very contextually informative to the language modelling task and contain a lot of useful information. It’s likely, then, that this information is also useful during the fine-tuning phase, to construct that lexical connection between input sentences and output classes. Moreover, when using the other two techniques, we focus on injected tokens that are often appended without suficient context to justify their presence at the end of the post. It may be that appending tokens to the end of the post may change the semantics of the sequence too much. The first method, which simply appends the label to the end of the post, often creates scenarios where the word appears to be out of place. The same applies to the second method, but thanks to the prompt, this efect is less noticeable. This efect may also be the reason why the first method is the worst performing one, while the Appended Label with Prompt method achieves F-Scores almost as high as the EOS one but without showing any useful correlation between the chosen representation and the model F-Score, thus not being usable as a Label Representation Selection method.
Figure 4 shows the variation in F-Scores obtained for each class. As we can observe, there is a quite
low degree of variance between the classes, in contrast with the results obtained in [
To have a deeper understanding of the efectiveness of the approach, we performed a more qualitative
analysis to determine which labels have the most substantial impact on the improvement or decline of
the classification accuracy. Table 3 reports the representations for each class obtained with the best and
worst performing models. As we can see, and in line with previous work [
To better understand the role of the representations frequencies in the training set we computed both the raw frequency of each representation in the whole dataset and the TF-IDF of the representations. We then calculated the Spearman Rank between the frequencies, the TF-IDFs, and the number of subtokens of the representations against the obtained F-Score and the Representation Sets rank the representations are in. The TF-IDF has been calculated by considering all the documents of a single category as a single document, and the documents’ length has been calculated as the total number of tokens (document lengths are reported in Appendix A). As we can observe (Table 4) representation frequency does not correlate to any class with the obtained F-Score. This, again, confirms that the role of the absolute frequency of a certain term in the training set doesn’t seem to have any positive or negative efect on the ability of the model to use such representation for its classes. However, by using the aggregation method that rewarded frequency mentioned in Sec. 2, we can see that for some classes the more frequent a word is, the more likely it is to be placed at a better rank. (Rank x Frequency column in Table 4). In particular, for two classes (SPORTS and AUTO-MOTO) more frequent representations had a higher chance to be placed in the best ranks. This could mean that the most informative words are frequently the same in these particular categories. Focusing on the TF-IDFs correlations, we can notice two negative statistically significant correlations with the F-score: SPORTS and ANIME. This is probably due to the fact that in-domain words, that don’t appear as often in the other categories, had a slightly positive impact on performances. Moreover, we noticed that these two categories are also those for which our model utilized several domain-specific words. In fact, the first ten ranked representations for the two categories are mostly domain-specific: • for SPORTS: campionato, gol, pareggio, centrocampo, milan, juventus, atalanta, tifosi, trequartista and derby; • for ANIME: streaming/download, grafio , manga, pokémon, pokemon, ko, morso, pokèmon, cmq, drago5.
Again, the correlation between the TF-IDF and the Representations’ rank was to be expected, based on the method we’ve used to aggregate the representations. We’ve also empirically noticed that the method we’ve used to extract the representations was keen on choosing domain-specific, low-frequency words. This is why often it chooses typos and similar words with errors in them. This is probably because low-frequency words are usually full words with much more semantic value in them, and by being domain-specific they carry high contextual information, useful for constructing the other tokens’ representations. This could explain why the TF-IDF, which is a metric that is specifically built to find such words, correlates so highly and significantly with the extracted words’ ranks.
Since Transformer models tokenize text by splitting them into subwords, we also tried to understand whether there is any correlation between both the F-score and the Representation rank with the number of subwords of the representations. From our results, we can see that subword length doesn’t seem to afect the model’s performance, nor does our selection technique seem to prefer words that are split into more or less subwords. The only two exceptions are AUTO-MOTO, where a higher number of subwords leads to a decrease in performances, and SPORTS, where our model seemed to place words with a higher subword number in lower places in the ranking system.
Finally, we investigated the impact of the Part-of-Speech (PoS) associated with the representations, both globally (See Figure 5) and on a per-class basis (Class-based distribution are reported in Appendix B). The PoS are extracted from an Italian Word Form Vocabulary developed by the Institute for Computational Linguistics (ILC) of the National Research Council of Italy (CNR), which contains all the word forms and their possible POs from the Italian language. As we can see from Figure 5, the most frequent PoS are UNKNOWN, VERB, NOUN, and ADJ. The class UNKNOWN contains the words that are not found in the Word Form Vocabulary, and these usually consist of typos, English words, proper nouns, etc. and are going to be seen more in detail for each category. The categories with the highest number of UNKNOWNs are ENTERTAINMENT, CELEBERITIES, ANIME, and SPORTS: 5morso, grafio and ko are all domain-specific words in the settings of the popular anime, cartoon and video-game Pokémon, with the first two being moves and the latter being a specific status.
• in ENTERTAINMENT, the majority of the UNKNOWNs are typos (e.g. cioe instead of cioè), abbreviations (e.g. nnt instead of niente), words with an increased vocal length in the last character (e.g. iniziaaaaaa instead of inizia) or english words (e.g. wish); • in CELEBRITIES, the majority are proper nouns (e.g. alessia, mirco, federica, etc.) and typos; • in ANIME, the majority are proper nouns of video games or tv shows characters (e.g. pokémon or charmender) or Japanese words (e.g. manga); • in SPORTS, the majority are proper nouns of soccer teams or players (e.g. milan, juventus, higuain, etc.) or match names composed by multiple teams or nation names (e.g. italia-uruguay or brasile-olanda) that our system didn’t split since they didn’t contain any spaces.
We also noted that for BIKES, NATURE, and AUTO-MOTO more VERBs are chosen instead of NOUNs, while for METAL-DETECTING, SPORTS, and SMOKE is the contrary. That being said, all the Parts-of-Speech seem reasonably distributed and it seems that no particular one is preferred by the method when choosing representations from the training set.
In this work, we presented a novel technique for reliably choosing label representation in
text-totext classification scenarios. This novel technique, based upon an Attention attribution technique
called Value Zeroing, provides a set of labels used to represent the class names for a text-to-text model.
We tested the approach on a Topic Classification task using IT5, an Italian pre-trained T5 model, by
training 100 diferent models with 100 sets of representation chosen this way. We found that choosing
representation with Value Zeroing and ranking them based on its value, leads to a useful correlation with
the trained model’s scores. Moreover, we noticed that choosing representation this way, leads to better
average performances and lower variance in performance, against both human- and random-chosen
representations [
We also conducted an in-depth analysis to understand whether either the performance of the model or our rankings were related to some simple statistics (frequency, TF-IDF, and the number of subtokens of the representations). Results showed some statistically significant correlations, especially when focusing on the TF-IDFs of the representations. We also found no interesting trend among the Parts-of-Speech of the representation chosen this way. While NOUNs and VERBs were the most popular, there weren’t any interesting findings, and some distributions suggest that the chosen representations are usually low-frequency in-domain words for that class.
In conclusion, our findings highlight again that the choice of label representations isn’t trivial and has an important impact on text-to-text classification performances and our technique seems to be a way to find a good solution for the label representation selection task.
Future research should focus on applying this technique to diferent kinds of tasks, primarily on those tasks where lexical and semantic clues from the text are essential in solving the task. Also, other aggregation methods should be tested, reducing the impact of the selection frequency, which showed not to be an important factor in the fine-tuned models’ performances.
of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 1688–1700. URL: https://aclanthology.org/2023.acl-long.94. doi:10.18653/v1/2023.acl-long.94. [14] H. Yamagiwa, S. Yokoi, H. Shimodaira, Improving word mover’s distance by leveraging selfattention matrix, in: H. Bouamor, J. Pino, K. Bali (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics, Singapore, 2023, pp. 11160–11183. URL: https://aclanthology.org/2023.findings-emnlp.746. doi: 10.18653/v1/2023. findings-emnlp.746. [15] Cimino, Dell’Orletta, Nissim, Tag-it – topic, age and gender prediction, EVALITA (2020). [16] H. Mohebbi, W. Zuidema, G. Chrupała, A. Alishahi, Quantifying context mixing in transformers, in: A. Vlachos, I. Augenstein (Eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Association for Computational Linguistics, Dubrovnik, Croatia, 2023, pp. 3378–3400. URL: https://aclanthology.org/2023.eacl-main.245. doi:10.18653/ v1/2023.eacl-main.245. [17] S. Mishra, S. Dutta, J. Long, D. Magazzeni, A survey on the robustness of feature importance and counterfactual explanations, 2023. URL: https://arxiv.org/abs/2111.00358. arXiv:2111.00358. [18] V. Basile, M. Di Maro, D. Croce, L. Passaro, Evalita 2020: Overview of the 7th evaluation campaign of natural language processing and speech tools for italian, in: 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop, EVALITA 2020, volume 2765, CEUR-ws, 2020. [19] A. Maslennikova, P. Labruna, A. Cimino, F. Dell’Orletta, Quanti anni hai? age identification for italian., in: CLiC-it, 2019. [20] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, C. Rafel, mT5: A massively multilingual pre-trained text-to-text transformer, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Online, 2021, pp. 483–498. URL: https: //aclanthology.org/2021.naacl-main.41. doi:10.18653/v1/2021.naacl-main.41. [21] K. Clark, U. Khandelwal, O. Levy, C. D. Manning, What does BERT look at? an analysis of BERT’s attention, in: T. Linzen, G. Chrupała, Y. Belinkov, D. Hupkes (Eds.), Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Association for Computational Linguistics, Florence, Italy, 2019, pp. 276–286.