The Human Value Detection task aims to identify the values humans express through words. Eight languages are included in the task, so the human value is meant to be identified in a multilingual context. Upon identification of the values, the value can be classified as having been attained, constrained, or neither attained nor constrained. The first subtask pertains to classifying a sentence to contain a specific human value. In contrast, the second subtask identifies whether the sentence constrains or attains the given set of human values [1].
We fine-tuned pre-trained RoBERTa and DeBERTa models based on the data from the ValuesML dataset. 1 We observed that the class imbalance in the data could affect the classification of the human values, so we attempted various combinations of upsampling the data to address this issue. We found that upsampling the lowest-performing categories by 4 folds yields the best results. Similar to other tasks (e.g., emotion detection [2]), we use only the English models and train using English and translated English utterances. Using the same model trained for both tasks, we were able to beat all the proposed baselines.
There are nineteen human values in Schwartz's Refined Theory of Basic Individual Values [3]. There is a wide range of values in this taxonomy. Additionally, since human values can be addressed implicitly, it can be challenging to identify which value is being used in a given text [4]. The human value detection task for 2023 had three components that made up each argument: a conclusion, a stance, and a premise.
Here is an example argument from the 2023 Human Values Task:
We should ban human cloning Stance in favor of Premise
We should ban human cloning as it will only cause huge issues when you have a bunch of the same humans running around all acting the same.
CLEF 2024: Conference and Labs of the Evaluation Forum, September 09-12, 2024, Grenoble, France ana72@cornell.edu (A. N. Aydin); ss2753@cornell.edu (S. Shaar); ctc9@cornell.edu (C. Cardie)
This year, the task takes a sentence as an input and outputs its human value, whether it is (even partially) attained, constrained, or neither. A value is attained if the text supports it and is constrained if it hinders it. This required us to treat the problem as a multi-class, multi-label classification problem with 38 labels consisting of 19 human values that are attained or constrained.
An example sentence with a human value is the following. The input is:
"Young women examining their options for third level education have been urged to consider careers in science, technology, engineering or maths (STEM). "
The value referred to in this sentence (the output) is Achievement Attained. These sentences are provided by the ValueML dataset , which contains text and their respective human values from articles and political text. data consists of 3000 texts in over eight languages. 20% of the data is used for validation, 20% is part of testing, and the remaining 60% is for training.
For all our experiments, we chose to combine all languages in the dataset into one by keeping the English utterances only. We use RoBERTa Large, specifically "FacebookAI/roberta-large" as the base model for our submission. We first experimented with the original RoBERTa and DeBERTa models but found the F1 scores on the validation dataset lower than the BERT baseline provided by the task organizers. We then fine-tuned RoBERTa-Large and DeBERTa-Large and observed improved results. There was an improvement, but the overall F1 score appeared to be affected by the class imbalance. Human Values that appeared less in the dataset had noticeably lower F1 scores than those that seemed more.
We attempted multiple configurations of upsampling but ended up using an upsampling technique where the human values with lower performance metrics were chosen to be upsampled by a factor of four.
To determine which human values to upsample, we fine-tuned the RoBERTa large model on the training data and analyzed the evaluation results. If the F1 score was 0.15 or less for subtask 1, we upsampled the attains and constrains form of that human value. We also looked at the metrics for subtask 2 to determine whether we should upsample only one of the attains or constrains of that value. If the recall for one of the constrains or attains was 50% or less than its counterpart, we only increase the underperforming one. For example, if the recall of the attains version of a human value was 50% or less than the constrains version, we only choose to upsample the attains version and vice versa. Based on the metrics and the unevenness of the dataset, we upsampled these values by a factor of four:
To fine-tune the model, we set the learning rate to 2e-5, used a warm-up ratio of 0.2, set the batch size to 8, and used four epochs. We put the random seed to 42, used the AdamW optimizer, and used a linear scheduler. We used one A100 GPU to run the experiments for the final submitted models. Our experiments used pre-trained RoBERTa [5] and DeBERTa [6] models. For evaluation, we use Precision, Recall, and the macro F1-score. We fine-tuned four models: RoBERTa and DeBERTa large on the upsampled data, and RoBERTa and DeBERTa large on the regular data, with no upsampling. For our final submission, we chose the model fine-tuned with the upsampled data using RoBERTa large as the base. This had the best metrics among all four models on the validation dataset. All four models can be found on Hugging Face2 .
Achieved F 1 -score of each submission on the test dataset for subtask 1. A ✓ indicates that the submission used the automatic translation to English. Baseline submissions shown in gray.
The upsampling methods resulted in a 4% increase for the macro F1 score of subtask 1 and a 2% increase for the macro F1 score of subtask 2. Of the upsampled human values, "Benevolence: caring" and "Tradition" were the only ones that did not have an improved F1-score for subtask 1. For the other upsampled values, there were improved F1-scores, some marginal and others significant. For subtask 2, the upsampled values performed only marginally better except for "Self-direction: thought" and "Humility", which all had worse metrics, and "Benevolence: dependability" and "Universalism: tolerance", which stayed the same.
After the task deadline, we looked at the performances of the other three models on the test data. Those models were DeBERTa large fine-tuned on upsampled data and DeBERTa large and RoBERTa large fine-tuned on the original training data. The results comparing all of these models are on Table 1 and Table 2. The RoBERTa large model with upsampling performs better than the other three models in subtask 1, but ends up being the worst-performing model on subtask 2. The best performing model for subtask 2 is DeBERTa large without upsampling.
We were able to beat the BERT baselines by incorporating the F1-Loss function and up-sampling lower-performing categories. This entails that the data benefits from knowledge transfer obtained from the different categories. In the future, we would also like to consider different languages as just using the translation could have led to missed cultural nuances expressed in language.