Human value detection consists in extracting human values from textual data. Being this a complex problem, the Semantic Evaluation 2023 workshop has dedicated one shared task, namely Task 4, to collect contributions and ideas on how to solve human value detection in arguments. This shared task has been organized as a challenge involving multiple teams, each of which have submitted an original solution. In this discussion paper, we present our team submission, reporting the system architecture employed and its performances1. By participating in SemEval 2023 Task 4, we noticed that none of the submitted solutions provide satisfying performances, hence we argue this task can still be considered an open issue. Therefore, we share the dificulties we experienced while trying to extract human values from arguments, and we provide a deep discussion on the types of error systems can make in this setting.
We should prohibit school prayer
against
it should be allowed if the student wants to pray as long as it is not interfering with his classes
[1, 1, 0, 0, . . . , 0, 1, 0, 0, . . . , 1, 0, 0, 0]
respective premise and stance. The target of classification is formulated as a vector = [
The main dataset is taken from the work by [
Listing 1: Value taxonomy example from json file.
Our contribution to SemEval Task 4 is an explainable value classification approach called SuperASKE: in this paper, we report the results achieved by our approach, discussing about its performances and providing a detailed error analysis. Furthermore, we want to open a discussion about complications and specificities characterizing human value detection. We argue that this task’s complexity is highly related to the nature of the target of classification, namely values categories, which can be considered as implicit information conveyed in arguments. Moreover, it is important to point out that recognizing values in texts is not a trivial task also for human beings.
This work is structured as follows: Section 2 gathers the relevant literature about human values and zero-shot learning classification systems; Section 3 gives details about the proposed system, SuperASKE, and the experimental settings we implemented for SemEval 2023 Task 4 submission; error analysis and discussion can be found in Section 5.
In this section we provide a brief background on human values definitions and schemes; moreover, we show some works done in the context of zero-shot learning (ZLS), which is the technique employed by our system SuperASKE to predict values from arguments.
ZSL is a problem setup in the field of machine learning, where a classifier is required to predict
labels of examples extracted from classes that were never observed in the training phase. It was
ifrstly referred to as dataless classification in 2008 [
ASKE [16] (Automated System for Knowledge Extraction) is our framework exploiting ZSL techniques and context-aware embeddings to extract semi-structured knowledge from textual corpora and collecting it in a graph-based data structure, called ASKE Conceptual Graph (ACG), whose nodes represent three kind of entities: document chunks (portions of the original documents), terms (n-grams find in the document chunks), and concepts (cluster of terms).
One of the components in ASKE is the embedding model used for computing the vectorial representations of the member of the ACG. In particular, we choose Sentence-BERT [17], a modification of BERT [ 18], that is specifically aimed at representing sentence meaning in a vector space. The model is capable of capturing the semantic aspect of the embedded tokens, since it computes context-aware embeddings which, in contrast with global ones, take into consideration the whole chunk of text in which they are used.
The framework we propose for detecting human values, namely SuperASKE (Figure 2), is a concatenation of ASKE and Random Forest [19] with the output of the former being employed as input for the latter. Every instance of this framework is tailored on a single human value, meaning that it solves a binary classification task. The main advantage of SuperASKE is the presence of two explainable models: in ASKE, concepts are described by the terms that compose them with the corresponding definitions; in the RF model, the same concepts are treated as feature, and the importance of each of them in the trees can be computed. This allows to identify which are the most influential concepts and how much they afect the final predictions.
Despite ASKE being a completely unsupervised model, running in a zero-shot setting, its lfexibility gives us the chance of proposing it in its supervised version tuned for classification. First of all, we proceed fine-tuning the sentence embedding model employed for computing the vector representation of the ACG entities (all-MiniLM-L6-v21). Being based on a siamese architecture, the model is fine-tuned by providing a pair of sentences and their corresponding semantic similarity. Therefore, we retrieve all the premises from the training set, and all the descriptions of the human values provided by the task organizers: if a premise is classified with a certain human value , all the possible pairs of and the descriptions of are given to the model with a similarity score of 1, otherwise the similarity is set to 0. The fine-tuned embedding model is then employed to compute the vector representation of the initial ACG components. Each human value is associated with its SuperASKE instance; within each SuperASKE instance, the ASKE component is initialized with only one concept, representing the respective human 1Model available at https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 value , associated with some dummy terms, having as definitions the ones provided in the value taxonomy. As document chunks , we consider only the premise of each argument, excluding stances and conclusions which appeared to not benefit to the final results. Afterwards, to ‘train’ ASKE, we run only on the premises positively classified as : in such a way, we ensure that the knowledge extracted by SuperASKE is relevant to the human value analyzed. Then, the ‘learned’ ACG is exploited to compute the similarities between the concepts occurring in it and the whole set of premises, now also including the ones negatively classified. The RF model is employed for the final binary classification. The matrix of similarities × , with = || and = ||, is provided as input for the model, together with the ground truth , determining if a premise is classified or not with a given human value. The trained RF model generates the ifnal predictions ˆ .
premises
For each human value , we retrieve only the set of premises classified as and we run diferent configurations of SuperASKE, changing the hyperparameters , and , with ∈ {− 1, ..., 0.5} being the similarity threshold for the zero-shot classification phase, ∈ {− 1, ..., 0.5} being the similarity threshold for the terminology enrichment phase, and ∈ {0, ..., 8} being the number of generations, i.e. the number of ASKE cycles performed. Each configuration learns a diferent version of the ACG, with its peculiar concepts and assigned terms. The diferent ACGs are then employed for computing the similarities between each premise in the entire dataset, considering also the one not classified as , and each concept occurring in the ACG. These similarities are used as input for the RF model, trained in order to predict the correct label for each premise w.r.t. the human value .
Based on the performances of the RF model on the validation set, we pick the best conifguration of hyperparameters for both the models of SuperASKE, ASKE and RF. We then proceed repeating the same steps for each human value, training 20 diferent binary classifiers. Evaluation is performed in two ways: using F1, precision and recall measures for each class independently and computing macro-averages over all categories. Oficial evaluations have been done on the TIRA platform [20].
F1 is placed near the median for the majority of the value categories. Though, this behaviour has some exceptions: “Self-direction: thought” and “Hedonism” F1 scores are in the first quartile, whereas “Power: resources”, “Stimulation” and “Humility” F1 scores are in the fourth quartile. Considering precision and recall it can be noticed that our model is, on average, less precise than the median precision score; however, SuperASKE has a higher recall, on average, than the median recall score.
Frequency-performance correlation. There is a positive correlation between models performance and value categories frequencies in datasets. Considering our model, the correlation ranges from a Pearson statistic of 0.51 with a pvalue of 0.019 (measured on the test-nahjalbalagha dataset), to a Pearson statistics of 0.72 with a pvalue of 0.00 (measured on the main test dataset). In this perspective, it is curious to see such a high F1 score for “Universalism: nature”. It can be hypothesized that this value category has a specific vocabulary, thus when nature-related words appear it is easier to guess the right class, both for humans and for automated models. Examples of words contained in “Universalism: nature” arguments are: “whaling”, “zoos”, “farming”, “human cloning”, “nuclear weapons”.
Confusion matrices. Our model’s recall is in the third quartile for most distributions (Figure 4): this information can be obtained also through the inspection of confusion matrices for each value category (see Table 3). As a symptom of this behaviour, it can be noticed that false positives frequencies tend to be higher than false negative frequencies. Furthermore, extracting some instances evaluated either as false positives or false negatives in the training set, it is possible to qualitatively categorize the type of error made by SuperASKE. Let’s consider the following argument, followed by its true positive and false positive values:
ID: E04080 Conclusion: We need an inclusive and pluralistic European society.
Stance: in favor of Premise: There need to be some rules for integration:Integration does not mean giving up European values and culture.
TP: Tradition
FP: Security: personal, Security: societal, Conformity: rules, Universalism: concern In this argument the main topic discussed is integration for immigrants. Some false positive value categories cannot be associated with this instance, but there could be two exceptions: actually, “Conformity: rules” could be associated with “rules for integration”, and “Universalism: concern” is about equality, which is a prerogative for integration measures. “Security: personal” and “Security: societal” are somewhat connected to immigration, but they are not directly involved in this argument. There are many other examples for which false positive value categories could make sense with the respective premises. This fact highlights how subjective and dificult is to label arguments with human values, thus training NLP models on this type of data leads to predictions for which it is hard to estimate a precise error measure. This problem arises also for true positive values which could be interpreted as false positives depending on the human annotator.
Value categories correlations. Comparing values categories correlations found in the ground truth and in the predicted labels, it is possible to study if and how SuperASKE learns correspondences among values. SuperASKE finds a higher number of correlated value-pairs than what is actually existing in the ground truth (7 pairs in predictions versus 3 pairs in true labels); moreover, correlations found in SuperASKE predictions are stronger than what can be seen in the ground truth (e.g. “Stimulation” and “Hedonism”).
Confounded value categories pairs. Pointing out which value categories are confounded the most with other classes can further explain why SuperASKE produces errors. With the term “confounded” we mean that a value category is false negative for an instance having another value category as false positive. However, as a matter of fact, there are not many instances for which SuperASKE confounds between value categories. In particular, the most confounded value categories for each dataset are: “Conformity interpersonal” and “Face” counting 9 confounded arguments in the validation set; “Face” and “Power: resources” having 13 confounded arguments in the training set. In the following we provide an example of confounding between “Power: resources” and “Face”.
ID: A07097 Conclusion: We should ban whaling Stance: against Premise: whaling is an important part of the diet, tradition and economies of many countries, and it is not our place to dictate terms as outsiders to their culture.
True label: Power: resources
False label: Face In this case, SuperASKE does not recognize the economical perspective of this argument; on the other hand, “whaling” can also be considered as a threat to national public images, depending on whether the opinion on this topic is positive or negative. This latter point of view could ofer a valid explanation to why this (and many others) argument is predicted with “Face” value category.
In this paper we present our solution to human value detection task, focusing on discussing issues about it. There are few points worth mentioning. First, we argue that a mechanism detecting implicit knowledge is needed. Personal human values could lead us to express a specific opinion about a discussed topic: however, arguments do not always embed the values or beliefs causing the stated opinion. For succeeding in solving tasks like human value detection, one must be able to extract causation paths linking implicit knowledge to expressed message. Second, one potential pitfall resides in the annotation procedure. Datasets used to train models are manually annotated. The problem arising here is that detecting human values in arguments is challenging also for humans. This leads to subjective annotations, which make the ground truth less reliable, and a lower-quality training procedure. We argue that a perspectivist approach in ground truthing would help modeling the subjectivity: this could translate into describing, in probability terms, if a certain value could be considered a ground truth label for an argument.
In conclusion, many improvements can be done to get closer and closer to achieving a good solution for human value detection. For instance, trying to map arguments to value categories though value categories descriptions could improve performances. Indeed, aligning value categories and argument terminology could help language models to produce better predictions.
⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆⋆ This project has received funding from the EU Horizon 2020 research and innovation programme under grant agreement No 101004949. This document reflects only the author’s view and the European Commission is not responsible for any use that may be made of the information it contains. [13] Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, B. Schiele, Latent embeddings for zero-shot classification, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 69–77. [14] X. Xu, T. Hospedales, S. Gong, Transductive zero-shot action recognition by word-vector embedding, International Journal of Computer Vision 123 (2017) 309–333. [15] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, T. Mikolov, Devise: A deep visual-semantic embedding model, Advances in neural information processing systems 26 (2013). [16] A. Ferrara, S. Picascia, D. Riva, Context-aware knowledge extraction from legal documents through zero-shot classification, in: Advances in Conceptual Modeling: ER 2022 Workshops, CMLS, EmpER, and JUSMOD, Hyderabad, India, October 17–20, 2022, Proceedings, Springer, 2023, pp. 81–90. [17] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, arXiv preprint arXiv:1908.10084 (2019). [18] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [19] L. Breiman, Machine Learning 45 (2001) 5–32. doi:10.1023/a:1010933404324. [20] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2023. A. Additional pictures and tables t h g u o h t : n o i t c e r i d f l e S n :iiil-ttfrcceeSaoodn iilttSaounm iseodnHm itceeevnhAm :irceeaoodPnnwm :ssrrrceeeoouPw ceaF :iltsrrceeSaoypun :iilttsrcceeSaoyu l a n o s r iitraodn :iltfsrreooyunm :iittfrreeooypnnm iiltyum :ilrcceeeeagvonnn T C C H B y t i l i b a d n e p e d : e c n e l o v e n e B
y :rcceonn :treaun :ltrceeaon :jiittcevbo iilssreavnm iilssreavnm iilssreavnm iilssreavnm U U U U Test set / Approach All
TP 1131 1204 1315 1255 1392 1392 1432 1517 1643 1614 1663 1712 1716 1726 1768 1764 1679 1755 1728 1771