Although personalization is a staple in several online settings, achieving an ad-hoc experience in some environments is impossible based on personal tastes. One such environment is the museum. In our view, visitors' facial reactions in front of artworks can play a crucial role. In this context, we want to study visitor behavior with an even finer-grained approach, identifying the most activated brain areas and how they relate to facial expressions. This paper describes how we intend to create a multimodal dataset to validate our study. We aim to fill a gap in personalizing the heritage experience with multidisciplinary research that combines neuroscience and computer science.
The personalization of the museum visitor experience dates back to 1997 when the “new museology” [1] was introduced, which diminished the centrality of the museum to give more importance to the visitor experience. Since then, visitors have become key interlocutors [2]. Despite this, it is not yet possible to appreciate true ad-hoc personalization for each visitor, with custom paths based on her preferences. Nowadays, an increasing number of museums are investing in this transition process1 and research is trying to bridge the gap [3, 4, 5, 6].
In this paper, we want to contribute to advancing research in museum experience personalization by studying the possibility of implicitly extracting user feedback from natural facial reactions. Although in the online domain the use of implicit feedback (e.g., clicks) to personalize the user experience has been a reality for several years (e.g., see [7]), the literature dealing with this topic in an ofline setting is still young [ 8, 9] and mainly limited to an analysis of expressions without any focus on the activation mechanisms behind a given facial expression. The ultimate goal of our research activities is to harness implicit feedback and data related to the museum visitors’ behavior [10] to enhance their cultural heritage experience [11] through, for instance, personalized itineraries [12] and multimedia material [13].
To go beyond a mere expression analysis, in this paper, we want to describe our research activities in which we employ electroencephalogram (EEG) signals to identify the most meaningful expressions and micro-expressions to predict user preferences during the fruition of artworks.
This section introduces some elements that help the reader to understand this multidisciplinary project.
The Circumplex Model of Emotions [14] is a fundamental psychological model for understanding and categorizing emotional responses. It classifies emotions according to two primary dimensions: valence captures the positivity or negativity, and arousal captures the intensity. In our context, this model will be used by testers to express explicit feedback regarding the artistic stimulus presented. The EEG is used to record spontaneous electrical activity of the brain. Applied to the study of user feedback and art appreciation, the EEG ofers the possibility of studying the cognitive processes associated with aesthetic experiences. This technology enables the Event-Related Potential (ERP) analysis, obtained as an average of several signals recorded in response to similar stimuli. These recorded responses provide information about the cognitive processing steps involved in stimulus recognition and processing. ERP helps to identify brain reaction times and regions involved in cognitive processes such as attention, memory, and perception [16]. Finally, we introduce the Facial Action Coding System (FACS) [18], which decomposes facial expressions into basic Action Units (AUs), which play a crucial role in emotion recognition. Using FACS, EEG data, and the Circumplex Model, we want to filter and identify participants’ most relevant facial expressions while experiencing the artwork. By doing so, we intend to objectively identify the most significant natural reactions for feedback detection from a neurological basis. EEG analysis is a well-known topic in the literature. Over the years, several datasets have been created to support diferent research ranging from motor tasks to sleep monitoring. Some noticeable datasets have allowed multimodal analysis of participant behavior, such as DEAP [20], SEED [21], DREAMER [22], and AMIGOS [23]. Despite these remarkable works, we decided to create a new dataset for several reasons. First, we wanted to study a particular stimulus set, such as artworks. Second, we wanted to use images rather than videos to fill a gap since all datasets use that media. Finally, we wanted to get more participants and stimuli with the assurance that each participant had all the videos available to analyze the correlations between the facial reactions and the traces of the brain’s electrical signals, given that in the datasets just described, the absence of some videos is not uncommon.
The study is voluntarily open to all Department of Engineering students at Roma Tre University. Participants must have normal or corrected vision using lenses or glasses. They must meet inclusion criteria, including signing a consent, absence of primary psychiatric and neurological conditions, and an age between 18 and 35. Exclusion criteria include substance or alcohol dependence, prior head trauma, insuficient knowledge of Italian, and current intake of certain medications that could afect cognitive abilities. Excluding those who meet these criteria will contribute to the study’s internal validity. Participants are asked not to consume cofee or alcohol in the two hours before the experiment.
The stimuli were identified through previous experimentation from 150 paintings rated as Positive (or Peaceful), Negative (or Disturbing), or Neutral by two domain experts. 512 students then evaluated the paintings by giving a rate from 1 to 7 regarding their likability. Finally, 33 paintings were randomly selected for each image category based on their mean value, assigned after the rating phase, to ensure a large stimulus spread that could elicit diferent emotions and reactions from the participants.
The experiment is carried out in the laboratory with controlled lighting and temperature, and the setting is designed to avoid aural or visual disturbances.
Two computers are used: the former is dedicated to presenting the stimuli, recording the videos, and saving the participants’ evaluations, and the latter is dedicated to recording the EEG. They are synchronized using a trigger box.
Specifically, the ActiCHamp device from BrainProducts GmbH connected to the Brain Recorder Vision 2.0 records the EEG trace. Cardiac parameters and muscle activity are detected simultaneously through electrodes attached to the Brain Recorder Vision 2.0. Video images are recorded using a dedicated video camera (i.e., Panasonic HC-VX870) controlled by USB. Stimulus images are presented on a 24-inch LACIE 324i screen, 1920x1200 resolution, and 10-bit gamma correction. The experiment script2 is written in Matlab using the Psychtoolbox3 [24] tool.
We randomly present one out of the 99 stimuli for three seconds to each participant. We use
a jittering of 0.15 seconds between each image, and each tester sees each image just once. In
the middle of the experiment, a two-minute break is given. During the image presentation, a
synchronization bit is sent to the EEG trace, with a value expressing the proposed stimulus
2github.com/LorenzoBattistiRomaTre/ScriptTesi.git (last access: October 23, 2023
type. After each stimulus, the participant has to answer the questions on arousal (
The assessment for personality and afective evaluation is carried out with the following surveys: • Personality assessment through the Big Five 10-question test [25]; • Alexithymia through the Toronto Alexithymia Scale [26], where alexithymia is defined as that personality disorder that impairs awareness and descriptive ability of experienced emotional states; • Emotion understanding and emotion regulation through the M-SCEIT Test [27]. It should be noted that these tests do not have any form of diagnostic purpose. They are only meant to check how the selected sample does not contain items with special conditions that could create bias in the results.
In this paper, we have described the idea and the experimental setting to collect data to study and analyze the possible correlation between facial reactions, EEG, and user preferences. Our ultimate goal is to shed light on potential pathways for personalization in the cultural heritage sector. Although the quest for true personalization in museums remains a complex challenge, we hope our study can represent a step toward the connection of art, neuroscience, and computer science.