This paper describes the submitted approach of Team Arthur Schopenhauer to Task 1 of the Touché lab at CLEF 2024. The goal of this task is twofold: detecting human values in texts (Subtask 1), and recognizing whether these values are attained or constrained (Subtask 2). The approach described in this paper simplifies Subtask 1 by restricting the detected values in a text to a maximum of one value. It also simplifies Subtask 2 by handling it separately from Subtask 1; that is, human values and attainment are detected independently of each other. This simplification strategy proved successful, as the submitted approach was ranked 2nd among the participating teams' best submissions (a single team can make multiple submissions) in Subtask 1 and was ranked 1st in Subtask 2. The described simplification results in two text-classification tasks, which are handled by fine-tuning and ensembling multiple BERT-based models.
The dataset provided by the organizers consists of the training set (labeled), the validation set
(labeled), and the test set (not labeled). It stems from the ValuesML project, itself part of a
broad JRC initiative that aims for a deep insight into values and identities [
The labeled part of the dataset contains zero-one labels for 19 human values, with two label columns for each value corresponding to constrained and attained, totaling 38 columns. A text may constrain or attain a specific value, but not both. However, there are texts where it is unclear whether the referenced value is attained or constrained, in which case both columns corresponding to the value are filled with 0.5.
The dataset contains texts from 9 languages. In addition, the organizers provide automated English translations for non-English texts. However, due to concerns regarding the accuracy of the translations, our approach uses the original texts and relies on multi-lingual language models.
Our submitted approach tackles the two described subtasks independently. Furthermore, the labeled datasets were divided into English and non-English texts, for each of which a diferent set of models was fine-tuned. Upon applying the fine-tuned models to the test set, which was used for the final submission, the texts in the test set were split in the same manner and the appropriate models were applied to each part.
This section describes how the original subtasks were transformed in order to simplify the model ifne-tuning process.
Subtask 1, in its given form, corresponds to a multi-label classification problem, because a single text may refer to multiple human values; that is, a single data instance may belong to multiple classes simultaneously. However, preliminary data analysis showed that approximately 94% of the labeled texts have either one label or no label. Therefore, for the sake of simplicity, it was decided to restrict the fine-tuning process to these instances, which turns the problem into a single-label classification problem. This simplification required introducing the no-label class for texts that have no label.
Subtask 2 was tackled independently of Subtask 1, which means the models of Subtask 2 were fine-tuned to predict a given text’s attainment, regardless of the human value that the text references. Accordingly, the simplified version of Subtask 2 corresponds to a single-label classification problem with two classes, namely attained and constrained.
The major steps of data preprocessing are shown in Figure 1. It begins by merging together the training and validation sets, then unuseful data is cleaned from the merged set, after which the dataset is reshaped to reflect the task simplification described in Section 3.1, and finally a new validation set is created within which the diferent strata of the labeled data are proportionally represented.
Original training Merge
set
Original validation set
Merged dataset
Filter out unuseful data
Reshape dataset
Create new split
New training set New validation set
After merging the training and validation sets, the following rows were removed from the merged
set with the help of the pandas [
To reflect the task simplification described in Section 3.1, the original 38 label columns were replaced by the following 2 columns: hv_value A numeric code for the human value referenced by the text (including no-label). attainment A numeric code for attainment (constrained, attained, or unknown). The unknown code is assigned to texts that do not have a human value label, or for which the attainment was unclear in the original dataset.
In addition, rows with the human value Humility were removed from the dataset. The reason for this additional filtering is that such rows are rare in the dataset and, after initial experiments, the fine-tuned models could not predict Humility with any accuracy.
The last step of data preprocessing was creating a new train-validation split. The validation set
was created using the proportional allocation strategy with a sampling rate of 0.1, whereby each
stratum is specified by a combination of language and label, for example all rows with language
“EN” and label “Conformity: interpersonal” form one stratum. Splitting was achieved using the
function train_test_split from scikit-learn [
For both subtasks, the approach relies on the pretrained models microsoft/deberta-v2-xxlarge
[
Training set Validation set
Language-based split
English training set English validation set Non-English training
set Non-English validation set
Fine-tuning
Pretrained deberta-v2-xxlarge
Pretrained xlm-roberta Fine-tuning
English fine-tuned
model Non-English finetuned model
For Subtask 1, bagging [
Our approach applies the commonly used cross-entropy loss function [
All ten models of our approach were fine-tuned using the same train-validation split, but with diferent hyperparameters, as described in Table 1. The remaining hyperparameters are described in Appendix A.
Fine-tuning was performed using PyTorch [
Ensembling is relevant only to Subtask 1, because for Subtask 2, only one model is used with each language subset.
Each of the models in Table 1 produces a predicted label2, along with a probability of that 1The Humility class was removed from the original 19 classes and the no-label class was added. 2For details on extracting predictions from neural network outputs, see https://www.learnpytorch.io/02_pytorch_ classification/ label. One common way to ensemble predictions is soft voting3. Our approach adjusts the original soft voting strategy by employing the concept of safe prediction, for want of a better term, which will be used to denote a prediction whose probability exceeds a certain threshold. With this definition of a safe prediction, ensembling was achieved using Algorithm 1 ( pruned soft voting). The rationale behind this algorithm is as follows: If one of the predictions is safe, while the others are not, then it should be chosen as the final prediction, regardless of the remaining predictions.
The threshold used in the final submission was obtained by repeatedly applying Algorithm 1 with a diferent threshold to the validation set and selecting the threshold that produced the best macro F1-score. For the English ensemble, the optimal threshold was 0.44, for the non-English ensemble: 0.49.
Table 2 displays a performance comparison between pruned soft voting and ordinary soft voting using the validation set and shows that pruned soft voting ofers a marginal improvement. However, it should be noted that, since the threshold for pruned soft voting was optimized using the validation set itself, the evaluation scores of pruned soft voting will be at least as high as those of soft voting, because soft voting is equivalent to pruned soft voting with threshold 0.
One point to consider when evaluating the ensembling strategies is that the no-label class (see 3.1.1) is included in the calculation of the F1-score (macro). This class will not be used in the ifnal evaluation of the approach by the organizers, so for each evaluation that we performed, a corresponding adjusted F1-score which does not include the no-label class was calculated. 3For details on soft voting, see https://machinelearningmastery.com/voting-ensembles-with-python/ Algorithm 1 Pruned Soft Voting Input:
Sequence of pairs ( 1, 1), . . . , ( , ) of predicted values for the label of one data instance, coupled with their prediction probabilities
Probability threshold Output:
One final label prediction if exists at least one probability in such that ≥ then return the final label prediction by applying soft voting only to those pairs ( , ) in with ≥ else
return the final label prediction by applying soft voting to the entire sequence end if
Upon submitting the approach, the fine-tuned models were applied to the test set and the results were exported to a .tsv file in the required format, which was submitted to the organizers. As texts with two words or less are believed to be noisy, the models of Subtask 1 were not applied to these texts; rather, no-label was manually predicted.
The evaluation results that were reported by the organizers are shown in Table 3 and Table 4. The reported F1-score for Subtask 1 (0.35) is significantly lower than the adjusted F1-score (0.3902) produced during our evaluation using the validation set (Table 2). This was expected for three reasons: 1. The the Humility label was not included when calculating the F1-scores in our evaluations.
Since our submission never predicts the Humility label, the F1-score for this label was 0 when our submission was evaluated by the organizers, thus reducing the overall macroaveraged F1-score. 2. The filtering described in Section 3.2.1 was not applied to the test set. In particular, the test set does contain texts with multiple labels, which our approach cannot cope with. 3. In the process of fine-tuning the models, the validation set was used for checkpointing; therefore, the models have a degree of overfitness to the validation set.
This paper presented the approach of Team Arthur Schopenhauer to Task 1 of the Touché lab at CLEF 2024. The main idea of the approach is simplifying the given subtasks. It simplifies Subtask 1 by eliminating the possibility of detecting multiple human values in a single text, and simplifies Subtask 2 by eliminating the possibility of detecting any dependence between referenced human values and attainment in a text. The source code for the approach is available under the following like: https://github.com/h-uns/clef2024-human-value-detection.
For future work, there are two notable areas of experimentation for improving the submitted approach: • Using larger or newer model architectures than the ones used in the approach. • Developing separate, specialized models that detect only certain subsets of human values, rather than all 19 values. Reducing the number of detectable human values is expected to improve the training eficiency of each model. In addition, combining such models can facilitate detecting multiple human values in a single text. ii:tttrceoogndhuh ii:i-ttrcceaoondn iltaon isnm teevnm :irceaodnnm :ssrrrceeou :iltsrreaoypn i:ilttsrceaoy iiton i:ltsrreoyum i:ilfttsrrreeaooypnnm ility l:ircceeaogvnn l:iiltceeeeaovynpdndb li:ssrrcceeaonnm li:tssrreeaaunm li:ltssrrceeeaaonm
EN llA lf-eS lfeS itSum eodH icehA eoPw eoPw ceaF ceuS ceSu radT fonC onC uHm eenB eenB ivnU ivnU ivnU Submission valueeval24-arthur-schopenhauer 83 77 83 85 88 87 73 84 80 82 84 78 80 79 74 91 89 86 85 81 valueeval24-bert-baseline-en ✓ 81 83 79 86 88 84 77 80 74 84 81 78 78 79 87 89 86 85 81 78 valueeval24-random-baseline ✓ 52 51 47 54 52 53 55 53 52 52 50 54 53 49 45 53 56 52 49 56
The approaches [