Introduction

Values motivate our actions [1] and impact all processes of our (moral) behaviour from perception and judgment to focus and action [2]. Being essentially the driving forces of individuals and societies, intelligibly identifying them empowers us to understand more profoundly, among others, our cultural heritage [3], citizen's political behaviour [4] and human interaction with artificial agents [5]. This knowledge can be fed back to people through the delivery of sustainable and responsible solutions from the related duty holders. Naturally, narratives are vessels of values. Historical texts, social media content, news items, and ChatGPT products are all resources to extract values and inform research, resolve sociopolitical tensions, deliver responsible AI. In particular, Touché [6] aims to advance our understanding of decision-making and opinion-forming processes by supporting the development of related methods and tools based on human values detection.

Human values detection in natural language is a complex task due to diverse perceptions, multilingualism, terminology interpretation, values attainment and actor attribution, among others. These are challenges that we have encountered through our research [7,8] and our participation in relevant projects 1,2 , and are also reflected in the performance of the models developed as part of SemEval-2023 Task 4: ValueEval [9]. Touché [6] at CLEF 2024 provides the opportunity to examine many of the aforementioned challenges. The provided dataset is a collection of 3000 human-annotated texts, including news articles and political texts, chosen to reflect diverse views. Over 70 people, from 9 language teams, annotated texts (in their mother tongue) for their value content and attainment. Schwartz's values [1] were adopted for the annotation vocabulary. In addition to the original files provided in nine languages, a machine-translated version of them in English was provided.

The current state of the art in Natural Language Processing (NLP) and ML/AI has enabled the development of methods that identify human values in natural language artifacts [10,11,12,13]. Advanced techniques for text classification, particularly for shorter text sequences, rely on fine-tuning Transformer-based models [7], Large Language Models (LLMs) [14], ensembles of Transformer-based models [15,16], and custom model architectures involving multiple heads with attention mechanisms [17,18].

This paper presents the methodology and results of Hierocles of Alexandria team in the Human Value Detection shared task of the second edition of Touché [6] at CLEF. Motivated by the availability of longer texts in this Touché edition and the availability of multilingual data, the innovative aspects of our approach include the modeling of contextual information, and the application of multi-task learning. Assuming that the value classification of a sentence may depend on earlier sentences and their classifications, previous sentences and their labels (either from annotated data during training or from classification of previous sentences during evaluation) are provided as input along with the sentence under classification. Multi-task learning, in the form of language-specific classification tasks, has been employed in order to capture potential different value instantiations in different languages.

Our approach leveraged fine-tuning four Transformer-based language models within a custom multitask with multiple heads model architecture, specifically tailored for multi-lingual and multi-label text classification in order to capture the linguistic nuances. Our experimental strategy comprised a comprehensive data analysis, the application of various loss functions, and the utilization of class positive weights to mitigate the challenge of class imbalance. Our approach achieved the highest score, securing the 1 𝑠𝑡 place for both multilingual and English submissions in sub-task 1, surpassing all other participating teams and baselines. The code for our approach is available on the provided GitHub link. 3In the context of Touché, the results from all approaches were submitted through the TIRA platform, which ensured the reproducibility and reliability of the software employed by participants, thereby facilitating the comparison of information retrieval experiments [19].

The structure of this paper is as follows: Section 2 explores the background. Several aspects of the data, including data analysis, pre-processing and an exploratory phase, are presented in Section 3. Section 4 introduces an overview of the developed system and the experiments. Section 5 presents the results. Finally, in Section 6 the conclusions are discussed, including limitations and future work.

Background

The exploration of human values within Natural Language Processing (NLP) encompasses various theoretical and empirical endeavors. Central to this exploration is Shalom H. Schwartz's theory of basic human values, which identifies nineteen universal values inherent to human behavior and cultural expression. These values, driven by distinct motivational goals, form a circular structure that illustrates their dynamic interplay, where pursuing one value may align with or conflict with another [1]. Schwartz's framework provides valuable insights into the motivational goals driving human actions and the complex interrelations among different values. In NLP, this framework offers a robust foundation for identifying and interpreting human values embedded within language.

In a significant research endeavor focused on identifying human values in NLP, a comprehensive taxonomy comprising 54 human values was crafted, aligning closely with psychological research. The researchers also introduced the initial annotated dataset for studying human values behind arguments [20]. This dataset encompassed 5.270 arguments from four distinct cultures: Africa, China, India, and the USA. Each argument in the dataset consisted of a premise, a conclusion, and a stance attribute indicating whether the premise supported or opposed the conclusion. The researchers manually annotated these arguments for human values. Their methodology has paved the way for automating the classification of human values, yielding promising results, with F1-scores reaching up to 0,81 and averaging 0,25, establishing a benchmark for future research in this domain.

To further advance the field of human values detection in argumentative texts, the authors of the aforementioned research organized the ValueEval: Identification of Human Values Behind Arguments shared task 4 in SemEval-2023 [9] by mapping the 54 human values from their previous research to a set of 20 value categories for multi-label classification. The task showcased both the potential and challenges associated with identifying human values in argumentative texts. A total of 39 teams contributed their methodologies, utilizing the Touché23-ValueEval Dataset comprising 9.324 arguments sourced from 6 diverse outlets, including religious texts, political forums, free-text arguments, newspaper editorials, and online democracy platforms in English [21]. Each argument included a premise, a conclusion, and a stance attribute signifying whether the premise was in favor of or against the conclusion. The teams' approaches were primarily evaluated on the Macro-F1 score.

The task's winner, the Adam-Smith team, achieved an F1 score of 0,56 by calculating a global decision threshold during training that optimizes the F1 score. They mainly employed twelve individual Transformer-based models that are ensembled in order to perform multi-label classification. [16]. The second-place John-Arthur team found that it is beneficial to encode the input data by adding tokenizer's special token separators, corresponding to low-cardinality values of Stance (in favour of vs against). Also, they fine-tuned larger Language Models, which performed better. Lastly, they adopted a threshold of 0,2 at the output of the sigmoid function to get the binary predictions for each human value, achieving an F1 score of 0,55 [14]. Addressing implicit value discrimination and data imbalance, the PAI team employed a multi-label classification model with a class-balanced loss function, securing multiple top positions across task categories with an overall average score of 0,54, placing them third [15]. The Mao-Zedong team's introduction of a multi-head attention mechanism and a contrastive learningenhanced K-nearest neighbor mechanism resulted in an F1 score of 0,53, placing them fourth [17]. Finally, certain members of the Hierocles of Alexandria team, who participated in that year's task as part of the Andronicus of Rhodes team, leveraged a Transformer model with four classification heads and applied two classification strategies with different activation and loss functions. In addition, they used two different data partitioning methods to handle class imbalance. Their system, employing majority voting, achieved an F1 score of 0,48, placing them in the upper half of the competition [7].

Inspired by the best methodologies employed in ValueEval, our approach aimed to tackle class imbalance and improve the classification performance, through the use of sigmoid threshold, larger language models, and tokenizer's special tokens in the encoded input. Nevertheless, Touché's Human Value Detection task at CLEF 2024 [6] has extended human value detection by integrating multiple languages, besides English. The introduction of new languages introduces more challenges, such as possible differences in annotation styles among languages, which adds complexity to the problem. To this end, our proposed approach was customised to the dataset features and the task by incorporating techniques that address the issue of multi-linguality and capture the linguistic nuances. The introduction of multiple languages and the need to address language-specific phenomena, was the main motivation behind our proposed approach in this paper, which includes a model architecture with multiple heads that are specifically tailored to each language, aiming to model more accurately multi-label text classification across multiple languages.

Data

Data Analysis

The dataset comprises 2.648 complete texts in nine languages: English, Greek, German, French, Bulgarian, Hebrew, Italian, Dutch, and Turkish. The dataset is split by the shared task organisers into training, validation and test sets: Of these texts, 1.603 are used for training, 523 for validation, and 522 for testing.

The number of annotated texts per language varies, as illustrated in Table 1. English has the highest number of texts (408), while French and Hebrew have the fewest (219 and 250, respectively). Each text is segmented into sentences, resulting in 74.231 sentences: 44.758 for training, 14.904 for validation, and 14.569 for testing.

The number of labels varies for each language, as shown in Table 1. There is no correlation between the number of texts and the number of labels. For instance, even though Hebrew texts were among the fewest, they had the highest number of labels (4.992).

For sub-task 1, the texts are annotated with Schwartz's 19 personal values [1]. Nearly half of the sentences (30.662 out of 59.662) are labelled with one or more values. Labels for the test set are not provided to evaluate participating systems. For sub-task 2, each classified value includes an annotation indicating whether the value is attained or constrained by the sentence, resulting in a final dataset with 38 classes.

The frequency of human value labels varies quite a bit, as depicted in Figure 1. Security: societal is the most frequently used value, with over 5.000 labels. Achievement and Conformity: rules are also quite popular, with over 3.000 labels each. On the other hand, Self-direction: thought, Universalism: tolerance, and Humility are less commonly used, with fewer than 1.000 labels. In fact, Humility is the least represented value, with only 151 labels.

All data was provided in the original language and translated using the DeepL API, except for Hebrew, which was translated using the Google Translate API. Figure 1: Labels' frequency across all texts. Some labels like "Security societal", "Conformity: Interpersonal" and "Achievement" are more frequent within the dataset.

Data Pre-processing

The dataset provides unique identifiers for each sentence, indicating the source text and its position within that text. Despite individual sentences being labeled independently, annotators were presented with the complete text for annotation, potentially influencing their assessments based on the overarching context.

To address this consideration, our pre-processing approach focuses on adding contextual information by integrating the previous part of the text and its annotated values. This was achieved through two main strategies: incorporating previous sentences and adding special tokens.

1. Incorporating Previous Sentences. For each sentence, we appended the two preceding sentences to each target sentence, thereby providing context from the specific text. If the total number of tokens exceeded the maximum allowed by our base model (maximum: 512), tokens were removed starting from the most distant sentence. If the sentence was the first sentence of a text, no preceding sentences were added. The "</s>" separator token linked the preceding and target sentences. 2. Adding Special Tokens. We implemented special tokens to represent each class, such as "<Security:

societal>" representing the value "Security: societal". We used the annotated labels from the previous two sentences for each sentence and appended them to the end. No special token was added if there was no annotated label for the previous sentences. This enables the classifier to interpret the annotator's perspective for better contextual understanding. These tokens were added as special tokens in the model's tokenizer and the token embedding matrix of the model was resized. They were assigned to attributes in the tokenizer for easy access and to make sure they were not split during tokenization. For predictions on the validation and test sets, the predicted classes were used as special tokens to enhance the model's contextual understanding.

The following is an example of pre-processed English text for the model input:

[CLS] Having spoken to many different left-leaning Hispanics, Avila said, "they are really beginning to feel like the Democratic party has become too extreme to the point where it's starting to scare some of them." <Security: societal> </s> Many are beginning to turn away from the Democratic party because "they're getting vibes of a communist Cuba and socialist Venezuela here in America. " As a result, Avila said Hispanics are going to be "extremely instrumental" in the upcoming midterm elections. <Self-direction: action> </s> "They are starting to come to the realization that their conservative values are in opposition to what the media has been trying to feed them in favor of Biden and the Democrats. " [SEP]

Exploratory Phase

We carried out several experiments to explore the behaviour of the pre-trained language models in order to form a baseline for our development process. This phase primarily exploits the language models to assess how well the collective human values of the mentioned dataset are captured by pretrained Transformer models [18], enhanced with classification heads so as to perform multi-label text classification, given the respective textual inputs. This process ensures that the models are fine-tuned to adequately fit the dataset with respect to all the language and sentence constraints.

The baseline experiments involved both multi-lingual (all languages together) and mono-lingual (each language separately) tests in order to record the effect of the special traits of each language on the human values. In general, we observed that the multi-lingual performance of the baseline models on the human value classification is higher than the performance on the individual languages in the mono-lingual experiments, as shown in Table 4 of Appendix A. This outcome could be explained by the close relation of the several inherent features (e.g context, vocabulary) of each language to the human value perception.

To further inspect this correlation and to note the bias of each language, we developed a more specific architecture that is built upon the baseline models, in order to improve the performance.

System Overview

Model Architecture

Based on the findings in Section 3.1, Figure 1 illustrates the imbalance in label distribution across different languages. This imbalance is partly due to the varying annotation styles among languages. For instance, some languages, such as English, have a large number of examples but fewer annotations, while others, like Hebrew, have fewer examples but many annotations. Consequently, similar sentences in different languages may receive different annotations. Experimental results indicate that a single out-of-the-box pre-trained Transformer model fails to effectively capture the unique linguistic features of each language, given the different annotation styles per language. In contrast, models that are fine-tuned for a specific language outperform those that are fine-tuned across all languages.

To address the multi-lingual nature of the problem and the differences in annotations between each language, a custom ensemble model was constructed. The architecture, as seen in Figure 2, leverages a pre-trained Transformer language model as its foundation. On top of this, nine custom Transformer heads were added, each tailored to a specific language: English, Greek, Dutch, Turkish, French, Bulgarian, Hebrew, Italian, and German.

Pretrained Base Model [CLS] 𝑇 −2 1 ... 𝑇 −2 𝑘 [𝑆𝑇 −2 ] 𝑇 −1 1 ... 𝑇 −1 𝑙 [𝑆𝑇 −1 ] 𝑇 1 𝑇 2 ... 𝑇 𝑚 [SEP]

Language Splitter 1. The input batch is fed into the pre-trained base model (Transformer).

Language

2. The output of the pre-trained model is passed through the language splitter which splits it according to the language identifiers within the batch. Each split tensor is directed to the corresponding custom Transformer head based on its language for further processing. 3. The logits produced by each custom Transformer head are concatenated into a single batch through the language combiner. 4. The concatenated logits batch is passed through the loss function to compute the training loss.

Model performs backpropagation.

This approach allows the model to handle multiple languages effectively by utilizing specialized components tailored to the linguistic features and annotation styles of each language.

Experimental Setup

The decision to utilize open-source Transformer-based multi-lingual models, specifically obtained from the Hugging Face platform, in our research was motivated by the multi-lingual composition of the data, which encompassed nine distinct languages. These models have been pre-trained across various languages, rendering them an optimal choice for providing a robust and comprehensive framework for analyzing and interpreting multi-lingual data. Consequently, we employed such models to ensure the effective capture of language-specific nuances and contexts, leading to more accurate and reliable results. For the data that underwent automatic translation into English, we employed open-source Transformer-based models that were exclusively pre-trained in English. This approach ensured an optimal understanding and interpretation of the nuances of the English language, thereby bolstering the accuracy of our analysis.

We utilized the multi-lingual base version of the RoBERTa Transformer-based language model [22], XLM-RoBERTa-base4 [23], which underwent pre-training on 100 languages with 768 layers. This model was employed to conduct preliminary experiments for multi-label text classification using AutoModelForSequenceClassification. Subsequently, baseline scores were obtained during the exploratory phase (See section 3.3). After analyzing the baseline results for individual languages and all languages collectively (see Table 4 in Appendix A), we leveraged the larger 1024-layer version, XLM-RoBERTalarge5 [23], to conduct further experiments involving loss functions, class weights, and different class thresholds (see sections 4.3 and 4.4). The purpose was to address the challenges of class imbalance and language disparities. These experiments were primarily facilitated using the Transformers and Hugging Face libraries, in conjunction with 2 NVIDIA TITAN RTX GPU cards, with 24GB VRAM each.

For both sub-tasks, we fine-tuned two Transformer-based language models for the multi-lingual data using the custom model architecture with multiple heads presented in section 4.1, with each head focusing on a specific language. The employed models were the XLM-RoBERTa-large [23] and the XLM-RoBERTa-xl6 [24], with 1024 and 2560 layers, respectively. In the case of the English-translated data, we integrated the RoBERTa-large7 [22] and the DeBERTa-v2-xxl8 [25] models, consisting of 1024 and 1536 layers, respectively, into the custom multi-head architecture, focusing solely on English for both sub-tasks.

We initially fine-tuned our models using the provided training data and fine-tuned them using the validation set. During the fine-tuning process, we established the hyperparameters, finalized the loss function, and determined the best thresholds for our submitted results. Then, we combined the training and validation data to use as the training set for fine-tuning, without having a separate validation set, using the previously defined hyperparameters. An overview of the hyperparameters used for our experiments and submissions is provided in Table 5 of the Appendix A.

As for the custom model architecture, the custom head for RoBERTa-large and XLM-RoBERTa-large included three Transformer layers, while the custom head for XLM-RoBERTa-xl employed only one. Due to GPU VRAM memory and time limitations, the DeBERTa-v2-xxl did not incorporate any Transformer layers in its custom head. The experiments with the custom model architecture, which form the final submissions, were conducted using 2 NVIDIA H100 PCIe GPU cards, with 80GB VRAM each.

Loss Functions & Class Weights

Various loss functions, including Binary Cross-Entropy Loss with Logits9 , Focal Loss [26], Classbalanced Loss [27], Distribution-Balanced Loss [28], and Class-balanced Negative Tolerant Regularization Loss [29], were tested by modifying the Trainer class from Hugging Face. These loss functions were employed as they were originally developed for handling data imbalance issues. They have previously been employed for the detection of human values by the PAI team in SemEval-2023 [15]. Positive weights were also calculated for each class to give more importance to the under-represented classes during model training, thereby improving the model's performance in these classes. The experiments using the XLM-RoBERTa-large with the standard classification head (AutoModelForSequenceClassification) showed that the Binary Cross-Entropy Loss with Logits achieved the best results. Therefore, this loss function was used for all the submitted runs with and without class positive weights.

Thresholds

Initial experiments were conducted with various thresholds ranging from 0,1 to 0,95. The Macro-F1 score for all classes and each class separately was calculated during fine-tuning and evaluating with the provided validation set. After applying the sigmoid function to the validation and test set predictions, the predictions were converted into 1 if they were equal to or higher than the threshold and 0 if they were lower than the threshold. Consequently, 3 separate prediction files were created based on the 0,5 default threshold, the best general threshold for all classes, and the best threshold for each class. Based on the results from the validation set, the prediction file utilizing the optimal threshold for each class demonstrated the highest scores. Therefore, all predictions submitted for the test set were generated by determining the optimal threshold for each class individually.

Results

Sub-task 1

The data presented in Table 2 illustrates that our test set submissions demonstrated significant improvement over the baseline scores in both the multi-lingual and translated English datasets. Among the multi-lingual multi-label multi-head models, the XLM-RoBERTa-xl model achieved the highest Macro-F1 score (39%) across all 38 classes by utilizing context and special tokens without class positive weights and being fine-tuned on the combined training and validation data as the training set. Conversely, the XLM-RoBERTa-large model, employing context, special tokens, class positive weights, and fine-tuned on 19 classes using only the training data, achieved the lowest score (34%).

In the context of the translated English data, the XLM-RoBERTa-large model, utilizing context and special tokens without class positive weights and having been fine-tuned on the combined training and validation data as the training set, produced the lowest Macro-F1 score across all classes (35%). At the same time, the remaining submissions yielded identical scores (37%).

Upon examining the F1 scores for each class individually, it becomes apparent that the Universalism: nature class achieved the highest F1 score at 63%, signifying successful detection by the models, as the remaining scores do not fall below 59%. Conversely, the classes with lower frequency in the texts were less accurately detected by the models. For instance, values such as Humility received a 0% F1 score in most submissions, with the highest score reaching only 11%. Furthermore, the models struggled to accurately classify the Self-direction: thought value, as their scores remained below 20%. Despite being one of the minority classes, the models correctly detected at least 27% of the annotated labels in the Universalism: tolerance class. The different model performance in classes is also evident in Figure 3 of the Appendix A, which illustrates the radar plot of the 19 values through the performance of our top-performing XLM-RoBERTa-xl model compared to the baseline models.

Sub-task 2

The data presented in Table 3 illustrates that our submission for the multi-lingual test dataset outperformed the baseline score. Utilizing the XLM-RoBERTa-xl for the multi-lingual dataset and the

Conclusion & Future Work

Our system, developed for Touché at CLEF 2024, addressed the "Human Value Detection" shared task by participating in both sub-tasks. This involved the fine-tuning of four Transformer language models within a custom multi-head model architecture for multi-label text classification. Our experimental approach encompassed the utilization of loss functions and class positive weights, as well as the incorporation of previous sentences as context and the representation of human values as special tokens. These measures were implemented to mitigate class imbalance and enhance the models' capacity to comprehend and classify texts more effectively. Our submissions demonstrated superior performance compared to the baseline and other participating teams' scores in both the multi-lingual and English-translated test datasets, resulting in achieving the 1 𝑠𝑡 place in sub-task 1. Despite scoring lower than the baseline in sub-task 2 in the English test dataset, our submission for the multilingual test dataset surpassed the baseline score. Notably, the XLM-RoBERTaxl model, leveraging context and special tokens without class positive weights and fine-tuned on the combined training and validation data, exhibited strong performance in both sub-tasks for the multilingual data. Furthermore, our findings indicated that while class positive weights augmented the models' ability to classify under-represented classes, they did not yield an overall performance improvement. The shared task posed a significant challenge due to the presence of data imbalance across classes and languages, as well as the existence of low-resource languages in the texts.

To further optimize model performance for multi-label human value detection, future endeavors should center on exploring additional Transformer layers within the custom multi-head architecture, with a particular emphasis on even larger Transformer language models such as the XLM-RoBERTa-xxl 10 . Additionally, the investigation of alternative loss functions to address data imbalance, the implementation of data augmentation methods or even an ensemble of various models hold the potential to further enhance performance.

Limitations

The experimentation process in both sub-tasks has revealed a significant issue of class imbalance. Despite the assignment of higher weights to the minority classes, it has become evident that detecting one or more human values is a challenging task. This challenge primarily stems from the imbalance in the annotated human values across languages as well as the general class imbalance among human values in the multi-lingual training dataset. Moreover, the presence of low-resource languages such as Hebrew and Greek has posed a further challenge, as the multi-lingual models contain a smaller number of tokens for these languages in comparison to English. Notwithstanding these challenges, the multilingual models have performed adequately compared to the baseline models. Moreover, in the process of fine-tuning the XLM-RoBERTa-xl and DeBERTa-v2-xxl models, we encountered challenges stemming from limitations in GPU VRAM memory and time. Specifically, we modified the fine-tuning approach for the first model by reducing the number of Transformer layers from three to one. Furthermore, in the second model's case, the custom head's multi-head architecture did not incorporate any Transformer layers.

A. Appendix Table 4

Achieved F 1 -score of the baseline XLM-RoBERTa-base on the validation dataset for sub-task 1. The model was fine-tuned using the whole multi-lingual training dataset and evaluated in multi-lingual validation dataset.

Languages