Emotions from Users' Annotations in Virtual Museums: a Case Study on the Pop-up VR Museum of the Design Museum Helsinki

Abstract

The paper presents a combined approach to knowledge-based emotion attribution and classification of cultural items employed in the H2020 EU project SPICE (Social cohesion, Participation, and Inclusion through Cultural Engagement)1. In particular, we describe an experimentation conducted on a selection of items contributed by the virtual museum (Pop-up VR Museum) of Finnish design objects, created by the Design Museum Helsinki in cooperation with the Aalto University. The results show an overlapping between the emotional labels extracted from the user-generated stories attached to the objects in the collection and the emotional annotations created by the audience during the virtual visit of the collection.

Keywords

• Museum

-

VR-Museum

1. Introduction

∗Corresponding author. †These authors contributed equally.

For centuries, the role of emotions in the experience of art has been acknowledged by aesthetics; only recently, however, the availability of tools for measuring human emotions at the physiological level has confirmed this intuition, showing that correlates of emotions, such as brain response and face expressions, are afected by the experience of art [ 1, 2 ]. In addition to their role in the way people relate to artworks [ 3 ], emotions provide a universal language through which people communicate their experience, well beyond words. An example of the capability of emotions to provide a universal means of expression is given by the difusion of emojis, widely used also by the communities of users who may have dificulties in producing written text, such as the d/Deaf [ 4 ]. Rooted in evolution, emotion are characterized by an universal basis [ 5 ], despite the diferences in their expression across languages and the cultures. In this sense, emotions can provide a way for connecting people who belong to diferent groups, intended as culture, age, education, and diferent sensory characteristics. The expression of emotions through language, in particular, lies at the basis of several models of emotions, including Shaver’s [ 6 ] and Plutchik’s [ 7 ], and has prompted the creation of a number of resources for sentiment analysis [ 8, 9, 10 ]. The application of these resources to art is straightforward: for example, WikiArt Emotions [11] is a dataset of 4,105 artworks from WikiArt annotated for the emotions evoked in the observer. The artworks were annotated via crowdsourcing for one or more of twenty emotion categories, in English language. Experiments such as WikiArt Emotions have paved the way to the extraction of emotions from text and tags to create afective art recommenders, like ArsEmotica [12] or DEGARI [13, 14, 15], able to classify and group artistic items well beyond the standard 6 basic emotions of Ekman’s theory [ 5 ], embracing richer, finer-grained models. A recent experiment on emotions evoked by art was performed in the Art Emotions Map project: 1,300 people were asked to describe how 1,500 paintings make them feel by choosing from diferent words. The results revealed 25 diferent emotions that people linked to the artworks they saw. The authors plotted these feelings on an interactive map, grouping artworks that triggered specific emotions.

In the context of the EU project SPICE (Social Participation and Inclusion through Cultural Engagement [16], which aims at supporting citizens in creating and sharing their own interpretation of artworks by attaching personal responses and afective annotations to artworks, our work has been focused on developing knowledge-based and reasoning technologies that leverage the role of emotions in the tasks of interpreting and reflecting on museums exhibits. In particular, we have developed a strategy to equip museum exhibits with emotional labels from user-generated comments that rests on the use of reasoning tools on a well-established emotion models (the Plutchik’s theory, described below). In this paper, the emotional labels have been derived from workshops and events wherein the participants listened to audio-recorded stories while testing the Pop-up VR Museum and in response, annotated their emotional feelings on a selection of items in the virtual museum (Pop-up VR Museum) in Helsinki. By applying our strategy on the user-generated contents in the Pop-up VR Museum, we obtained a fine-grained, comprehensive account of the emotions evoked by the items in the collection.

2. The Plutchik’s Ontological Model

The reference theory for the two systems is encoded in an ontology of emotional categories based on Plutchik’s psychological model of emotions [17]. The ontology structures emotional categories in a taxonomy, which currently includes 32 emotional concepts. The design of the taxonomic structure of emotional categories, of the disjunction axioms and of the object and data properties mirrors the main features of Plutchik’s circumplex model. As mentioned before, such model can be represented as a wheel of emotions (see Figure 1) and encodes the following elements: • Basic or primary emotions: Joy, Trust, Fear, Surprise, Sadness, Disgust, Anger, Anticipation; in the color wheel, this is represented by diferently colored sectors. • Opposites: basic emotions can be conceptualized in terms of polar opposites: Joy vs

Sadness, Anger vs Fear, Trust vs Disgust, Surprise vs Anticipation. • Intensity: each emotion can exist in varying degrees of intensity; in the wheel, this is represented by the vertical dimension. • Similarity: emotions vary in their degree of similarity to one another; in the wheel, this is represented by the radial dimension. • Complex emotions: a complex emotion is a composition of two basic emotions; the pair of basic emotions involved in the composition is called a dyad. Looking at the Plutchik wheel, the eight emotions in the blank spaces are compositions of similar basic emotions, called primary dyads. Pairs of less similar emotions are called secondary dyads (if the radial distance between them is 2) or tertiary dyads (if the distance is 3), while opposites cannot be combined.

Within this ontology, the class Emotion is the root for all the emotional concepts. The Emotions hierarchy includes all the 32 emotional categories as distinct labels. In particular, the Emotion class has two disjoint subclasses: BasicEmotion and ComplexEmotion. Basic emotions of the Plutchik model are direct sub-classes of BasicEmotion. Each of them is specialized again into two subclasses representing the same emotion with weaker or stronger intensity (e.g. the basic emotion Joy has Ecstasy and Serenity as sub-classes). Therefore, we have 24 emotional concepts subsumed by the BasicEmotion concept. Instead, the class CompositeEmotion has 24 subclasses, corresponding to the primary (Love, Submission, Awe, Disapproval, Remorse, Contempt, Aggressiveness e Optimism), secondary (Hope, Guilt, Curiosity, Despair, Unbelief, Envy, Cynicism e Pride) and tertiary (Anxiety, Delight, Sentimentality, Shame, Outrage, Pessimism, Morbidness, Dominance) dyads. Other relations in the Plutchik model have been expressed in the ontology by means of object properties: the hasOpposite property encodes the notion of polar opposites; the hasSibling property encodes the notion of similarity and the isComposedOf property encodes the notion of composition of basic emotions.

3. DEGARI

The core component of DEGARI relies on a probabilistic extension of a typicality-based Description Logic called TCL, (Typicality-based Compositional Logic, introduced in [18]). This framework allows one to describe and reason upon an ontology with commonsense (i.e. prototypical) descriptions of emotional concepts, as well as to dynamically generate novel prototypical concepts in a knowledge base as the result of a human-like recombination of the existing ones [19, 20, 21, 22, 23, 24].

The logic TCL, that we recall here for self-containedness, is the result of the integration of two main features: (i) an extension of a nonmonotonic Description Logic of typicality ℒ + TR introduced in [25] with a distributed semantics; (ii) a well-established heuristics inspired by cognitive semantics for concept combination and generation [26], in order to formalize a dominance efect between the concepts to be combined: for every combination, it distinguishes a HEAD, representing the stronger element of the combination, and a MODIFIER. The basic idea is to extend an initial knowledge base (ontology) with a prototypical description of a novel concept, obtained by the combination of two existing ones, namely a HEAD concept and a MODIFIER concept. In this logic, typical properties can be directly specified by means of a typicality operator T enriching the underlying Description Logic, and a knowledge base can contain inclusions of the form ∶∶ T() ⊑ to represent that “typical s are also ”, where is a real number between 0.5 and 1, representing the probability of finding elements of being also . From a semantic point of view, it considers models equipped by a preference relation among domain elements, where < means that is “more normal” than , and that the typical members of a concept are the minimal elements of with respect to this relation. An element is a typical instance of a given concept if belongs to the extension of the concept , written ∈ ℐ, and there is no element in ℐ “more normal” than . TCL also considers the key notion of scenario. Intuitively, a scenario is a knowledge base obtained by considering all rigid properties as well as all ABox facts, but only a subset of typicality properties. To this aim, it considers an extension of the Description Logic ℒ + TR based on the distribution semantics known as DISPONTE [27]. The idea is to assume that each typicality inclusion is independent from each other in order to define a probability distribution over scenarios: roughly speaking, a scenario is obtained by choosing, for each typicality inclusion, whether it is considered as true of false. Reasoning can then be restricted to either all or some scenarios. TCL equips each scenario with a probability, easily obtained as the product, for each typicality inclusion, of the probability in case the inclusion is involved, (1 − ) otherwise. It immediately follows that the probability of a scenario introduces a probability distribution over scenarios, that is to say the sum of the probabilities of all scenarios is 1.

The bridge from the definition of the emotions in the ontology and the annotations associated with an artwork is provided by an emotion lexicon. Emotional concepts are described by using the NRC Emotion Intensity Lexicon [ 8 ] (one of the lexica used also by SOPHIA). Such lexicon provides a list of English words, each with real-values representing intensity scores for the eight basic emotions of Plutchik’s theory. The lexicon contains close to 10, 000 words, including terms already known to be associated with emotions as well as terms that co-occur in Twitter posts that convey emotions. The intensity scores were obtained via crowdsourcing, using best-worst scaling annotation scheme. In this work, we considered the most frequent terms available in such lexicon (and associated to the basic emotions of the Plutchik wheel) as typical features of such emotions. In this way, once the prototypes of the basic emotional concepts were formed, the TCL reasoning framework was used to generate the compound emotions.

In the context of our system, TCL allows us to provide a formal, explainable framework for combining prototypical descriptions of concepts. It is adopted to automatically build the prototypical representations of the compound emotions according to the Plutchik’s theory. The prototypes of basic emotions are formalized by means of a TCL knowledge base, whose TBox contains both rigid inclusions of the form

BasicEmotion ⊑ Concept,

to express essential desiderata but also constraints, e.g. Joy ⊑ PositiveEmotion as well as prototypical properties: ∶∶

T(BasicEmotion) ⊑ TypicalConcept,

representing typical concepts of a given emotion, where ∈ (0.5, 1] , expressing the frequency of such a concept in items belonging to that emotion: for instance, 0.72 ∶∶ T(Surprise) ⊑ Delight is used to express that the typical feature of being surprised contains/refers to the emotional concept Delight with a frequency/probability/degree of belief of the 72%.

Once the association of lexical features to the emotional concepts in the Plutchik’s ontology is obtained and the compound emotions are generated via the logic TCL, the system is able to reclassify the artworks in the novel emotional categories. Intuitively, an item belongs to the new generated emotion if its metadata (name, description, title, user-generated annotations) contain all the rigid properties as well as at least the 30% of the typical properties of such a derived emotion. The 30% threshold was empirically determined: i.e., it is the percentage that provides the better trade-of between overcategorization and missed categorizations [ 28].

4. Analyzing the Stories from Pop-up VR-Museum

The Pop-up VR Museum is a collaborative endeavor between Aalto University and Design Museum Helsinki (DMH), resulting in a prototype of a Virtual Reality (VR) application that provides citizens with access to a digitized collection of artifacts from DMH. Some of the features ofered by the application include: • Interactive Object Manipulation: Users have the capability to interact with virtual design objects, enabling actions such as picking them up, rotating them, and even altering their color variants within the VR environment. • Engagement with narratives: Users are encouraged to contribute their own stories, while also having the opportunity to read and listen to narratives shared by other contributors. Additionally, users can annotate these stories using a range of emotions, represented through emojis. • Immersive Environments: Each virtual object is accompanied by a distinct virtual environment, allowing users to immerse themselves and gain fresh perspectives on the object. This VR experience employs a Natural User Interface (NUI) to facilitate user interaction, characterized by its minimal interaction requirements (Reference needed here). This design choice results in a gentle learning curve for users, potentially enhancing their sense of presence within the VR environment as well (References needed).

In the Pop-up VR Museum, the functionality for annotating emotions becomes available to users exclusively after they have thoroughly engaged with the content. Specifically, this feature is activated once a user has completed the process of listening to a narrative and immersing themselves in the design object. This criterion was deliberately chosen in the case study to guarantee that users experience the entirety of a story without any premature interruptions. Moreover, it ensures that users have the opportunity to engage with the object from a novel perspective within the VR environment. It is only under these circumstances that a user is deemed eligible to select and annotate their emotions to a particular story.

Before the integration of DEGARI 2.0 emotions for user annotation, the initial iterations of the prototype employed a Likert scale range (1-5) of commonly used emojis, as typically seen in surveys for gathering user feedback. This range encompassed the following emotions, ranging from sadness to happiness: , , , , and . Notably, these emojis were not explicitly labeled for users, leaving them open to individual interpretation. Users were limited to selecting a single emoji, and the experience would proceed accordingly.

In the subsequent iterations, a more nuanced approach was introduced by ofering users the ability to annotate their emotions using the DEGARI 2.0 framework, which comprises nine distinct and complex emotional states. Users were presented with a specific question: ”How did the previous story make you feel? (Select multiple)”, accompanied by the option to choose multiple emotions that resonated with them. In order to select emojis that closely aligned with each DEGARI 2.0 emotion, the following were chosen: • Delight: • Love: • Joy: • Optimism: • Hope: • Curiosity: • Disapproval: • Anxiety: • Outrage: • Pessimism: • Shame:

However, it remains a subject of ongoing inquiry whether these emojis accurately encapsulate the full spectrum of DEGARI 2.0 emotions, especially in light of the potential language variations introduced by users selecting either Finnish or Swedish for their Pop-up VR Museum experience. In such cases, the selected texts would subsequently be translated into the chosen language, further emphasizing the need for continued evaluation and refinement.

In the broader context, the selection of emotions is influenced by a diverse array of factors: 1. User’s Emotional Response to the Story: The emotions chosen are contingent upon how the user personally interprets and reacts to the narrative presented. 2. Content Characteristics in the Virtual Environment: The emotional selection may also be influenced by the nature of the content within the virtual environment that the user is immersed in. 3. Sense of Presence in the Experience: The degree to which users feel present in the virtual environment is a critical determinant. Any disruptions in this sense of presence may lead to users expressing frustration and subsequently annotating negative emotions to stories. 4. Physical Surroundings during the VR Experience: Users engaging with the Pop-up VR Museum while immersed in VR may also be impacted by their immediate physical environment. This is particularly pertinent as many users often engage in conversations with mediators located in the real space, potentially influencing their emotional annotations.

The fist experiment conducted with DEGARI 2.0 was on the stories of the virtual museum (Pop-up VR Museum) in Helsinki (the used dataset is on the Linked Data Hub (LDH) 1 and has been used also for further analysis by the Thematic and Value Reasoners). In this case the term story refers to the users generated contents on specific artifacts. In particular, the first part of Table 1(a), shows some statistics calculated from the use of our DEGARI 2.0 system for the extraction of emotions on user-generated comments. The set of tuple of emotions that have been extracted by DEGARI 2.0 most frequently are {Delight, Joy}, {Love, Optimism} and {Love, Joy}. Table 1(b) shows for each extracted emotion from user-generated comments on stories, the relative frequency. The emotion that DEGARI 2.0 extracted most frequently, was Joy with 36 followed by Delight with 25.

The second experiment was to validate DEGARI 2.0 as a cognitive tool to investigate the true emotional content that is present within the stories. In particular, we wanted to understand how the emotions extracted from DEGARI 2.0 could be aligned with the emotions chosen by the users during the creation of their stories. In particular, the fist column shows the emotions extracted by DEGARI 2.0: the emotion ”Love” was extracted 19 times out of a total of 20 objects (95%), 15 times DEGARI 2.0 extracted the emotion ”Optimism” (75%), 9 times the emotion ”Hope” (45%), 4 times ”Joy” and ”Delight” (20%). The 4th column shows the number of users who chose a particular emotion related to the 20 objects. Specifically, 17 users chose ”Love”, 14 ”Optimism”, 4 the emotion ”Hope”, 8 the emotion ”Joy” and 7 users the emotion ”Delight”. Finally, the last column shows the perfect match between the emotions extracted by DEGARI 2.0 and the emotions selected by the users. In particular, 17 users have chosen the emotion ”Love” (85% of them had a perfect match with the 19 extracted by DEGARI 2.0). These results are shown in Table 2 and Table 3. 1https://spice.kmi.open.ac.uk/dataset/details/104

Total stories Total stories with extracted emotions Total extracted emotions by DEGARI 2.0 Mean extracted emotions foreach story MIN extracted emotions for each item MAX extracted emotions for each item Emotion Joy Delight Love Optimism Curiosity Pessimism Disapproval Hope Anxiety Shame Outrage (a) (b)

Frequency

5. Discussion and Conclusion

In this paper, we described the results of the analysis on the stories generated by the users for the design artefacts exhibited in the Pop-up VR Museum created by the Aalto University and Design Museum Helsinki (DMH) using the afective-based sensemaking system called DEGARI 2.0. The results of the analysis show that there is an overlapping between the emotional labels extracted from the user-contributed stories and the emotional labels added by other users to the same objects as part of their experience with the Pop-up VR-Museum. Although the sentiment of the user contributions (stories and annotations) is generally oriented towards positive emotions – due to the very same natures of the experience, which triggers a reflection on the personal engagement with the artefacts –, we think that data extracted by DEGARI 2.0 complement the DEGARI 2.0 emotions Love Optimism Joy Delight Hope

DEGARI 2.0 % on Extracted emotions total foreach object objects

Emotions selected by users users’ annotation in a way that can provide the museum curators with an useful insight on the emotional response of the audience expressed. In particular, the overlapping between the two types of data suggests that a basic human mechanism for the creation of social cohesion, namely emotional empathy, is at work also in the experience of art, paving the way to an ethical use of emotions to build cohesion. International Conference on Information and Knowledge Management, Ireland, October 19-23, 2020, 2020, pp. 105–114. [10] Y. Susanto, A. G. Livingstone, B. C. Ng, E. Cambria, The hourglass model revisited, IEEE

Intelligent Systems 35 (2020) 96–102. [11] S. Mohammad, S. Kiritchenko, WikiArt emotions: An annotated dataset of emotions evoked by art, in: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), European Language Resources Association (ELRA), Miyazaki, Japan, 2018. URL: https://www.aclweb.org/anthology/L18-1197. [12] V. Patti, F. Bertola, A. Lieto, Arsemotica for arsmeteo. org: Emotion-driven exploration of online art collections, in: I. Russell, W. Eberle (Eds.), The Twenty-Eighth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2015), Association for the Advancement of Artificial Intelligence, AAAI Press, 2015, pp. 288–293. [13] A. Lieto, G. L. Pozzato, S. Zoia, V. Patti, R. Damiano, A commonsense reasoning framework for explanatory emotion attribution, generation and re-classification, Knowl. Based Syst. 227 (2021) 107166. doi:10.1016/j.knosys.2021.107166. [14] A. Lieto, M. Striani, C. Gena, E. Dolza, A. M. Marras, G. L. Pozzato, R. Damiano, A sensemaking system for grouping and suggesting stories from multiple afective viewpoints in museums, Human-Computer Interaction 0 (2023) 1–35. doi:10.1080/07370024.2023. 2242355. [15] A. Lieto, G. L. Pozzato, M. Striani, S. Zoia, R. Damiano, Degari 2.0: A diversity-seeking, explainable, and afective art recommender for social inclusion, Cognitive Systems Research (2022). [16] E. Daga, L. Asprino, R. Damiano, M. Daquino, B. D. Agudo, A. Gangemi, T. Kuflik, A. Lieto, M. Maguire, A. M. Marras, et al., Integrating citizen experiences in cultural heritage archives: requirements, state of the art, and challenges, ACM Journal on Computing and Cultural Heritage (JOCCH) 15 (2022) 1–35. [17] R. Plutchik, A general psychoevolutionary theory of emotion, in: Theories of emotion,

Elsevier, 1980, pp. 3–33. [18] A. Lieto, G. Pozzato, A description logic framework for commonsense conceptual combination integrating typicality, probabilities and cognitive heuristics, Journal of Experimental and Theoretical Artificial Intelligence 32 (2020) 769–804. URL: https://doi.org/10.1080/ 0952813X.2019.1672799. doi:10.1080/0952813X.2019.1672799. [19] L. Giordano, V. Gliozzi, A. Lieto, N. Olivetti, G. L. Pozzato, Reasoning about typicality and probabilities in preferential description logics, Applications and Practices in Ontology Design, Extraction, and Reasoning, arXiv preprint arXiv:2004.09507 (2020). [20] A. Lieto, Cognitive design for artificial minds, Routledge, 2021. [21] A. Lieto, F. Perrone, G. L. Pozzato, E. Chiodino, Beyond subgoaling: A dynamic knowledge generation framework for creative problem solving in cognitive architectures, Cognitive Systems Research 58 (2019) 305–316. [22] A. Lieto, G. L. Pozzato, Applying a description logic of typicality as a generative tool for concept combination in computational creativity, Intelligenza Artificiale 13 (2019) 93–106. [23] A. Lieto, G. L. Pozzato, F. Perrone, E. Chiodino, Knowledge capturing via conceptual reframing: A goal-oriented framework for knowledge invention, in: Proceedings of the 10th International Conference on Knowledge Capture, 2019, pp. 109–114. [24] A. Lieto, G. L. Pozzato, A. Valese, M. Zito, A logic-based tool for dynamic generation and classification of musical content, in: International Conference of the Italian Association for Artificial Intelligence, Springer, 2022, pp. 313–326. [25] L. Giordano, V. Gliozzi, N. Olivetti, G. L. Pozzato, Semantic characterization of rational closure: From propositional logic to description logics, Artificial Intelligence 226 (2015) 1–33. URL: https://doi.org/10.1016/j.artint.2015.05.001. doi:10.1016/j.artint.2015.05. 001. [26] J. A. Hampton, Inheritance of attributes in natural concept conjunctions, Memory &

Cognition 15 (1987) 55–71. [27] F. Riguzzi, E. Bellodi, E. Lamma, R. Zese, Reasoning with probabilistic ontologies, in: Q. Yang, M. J. Wooldridge (Eds.), Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, AAAI Press, 2015, pp. 4310–4316. URL: http://ijcai.org/Abstract/15/613. [28] E. Chiodino, D. D. Luccio, A. Lieto, A. Messina, G. L. Pozzato, D. Rubinetti, A knowledgebased system for the dynamic generation and classification of novel contents in multimedia broadcasting, in: Proceedings of ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, 2020, volume 325 of Frontiers in Artificial Intelligence and Applications , IOS Press, 2020, pp. 680–687. URL: https://doi.org/10.3233/FAIA200154. doi:10.3233/FAIA200154.

References

[1] N. N. Van Dongen , J. W. Van Strien , K. Dijkstra , Implicit emotion regulation in the context of viewing artworks: ERP evidence in response to pleasant and unpleasant pictures , Brain and Cognition 107 ( 2016 ) 48 - 54 .

[2] H. Leder , G. Gerger , D. Brieber , N. Schwarz , What makes an art expert? Emotion and evaluation in art appreciation , Cognition and Emotion 28 ( 2014 ) 1137 - 1147 .

[3] I. Schindler , G. Hosoya, W. Menninghaus , U. Beermann , V. Wagner , M. Eid , K. R. Scherer , Measuring aesthetic emotions: A review of the literature and a new assessment tool , PloS one 12 ( 2017 ) e0178899 .

[4] K. Mack , D. Bragg , M. R. Morris , M. W. Bos , I. Albi , A. Monroy-Hernández , Social app accessibility for deaf signers , Proceedings of the ACM on Human-Computer Interaction 4 ( 2020 ) 1 - 31 .

[5] P. Ekman , W. V. Friesen , Constants across cultures in the face and emotion ., Journal of personality and social psychology 17 ( 1971 ) 124 .

[6] P. Shaver , J. Schwartz , D. Kirson , C. O'connor, Emotion knowledge: further exploration of a prototype approach ., Journal of Personality and Social Psychology 52 ( 1987 ) 1061 .

[7] R. Plutchik , The nature of emotions, American scientist 89 ( 2001 ) 344 - 350 .

[8] S. Mohammad , Word afect intensities , in: Proceedings of the Eleventh International Conference on Language Resources and Evaluation , LREC 2018 , Miyazaki, Japan, May 7- 12 , 2018 , European Language Resources Association (ELRA), 2018 . URL: http://www. lrec-conf.org/proceedings/lrec2018/summaries/329.html.

[9] E. Cambria , Y. Li , F. Z. Xing , S. Poria , K. Kwok , Senticnet 6: Ensemble application of symbolic and subsymbolic ai for sentiment analysis , in: CIKM '20: The 29th ACM