Current image/video acquisition and analysis techniques allow for not only the identification and classification of objects in a scene but also more sophisticated processing. For example, there are video cameras today able to capture micro facial expressions, namely, facial expressions that occur in a fraction of a second. Such micro expressions can provide useful information to define a person's emotional state. In this article, we propose to use these features to collect useful information for designing and implementing increasingly effective interactive technologies. In particular, facial micro expressions could be used to develop interfaces capable of fostering the social and cultural inclusion of users belonging to different realities and categories. The preliminary experimental results obtained by recording the reactions of individuals while observing artworks demonstrate the existence of correlations between the action units (i.e., single components of the muscular movement in which it is possible to break down facial expressions) and the emotional reactions of a sample of users, as well as correlations within some homogeneous groups of testers.
Systems capable of identifying a user's emotional state starting from her behavior are becoming more and more popular [1]. Among these, Automatic Facial Expression Analysis (AFEA) [2] systems are of particular importance. Facial expressions can be defined as facial changes in response to a person's internal emotional states, intentions, or social communications [3]. This research topic is certainly not new if we consider that Darwin in 1872 had already addressed the subject in [4]. Since then, there have been several attempts by behavioral scientists to conceive methods and models for the automatic analysis of facial expressions on image sequences [5,6]. These studies have laid the foundations for the realization of computer systems able to help us understand this natural form of communication among human beings (e.g., see [7,8,9,10]). Such systems, although very efficient, are inevitably affected by context, culture, genre and so on [11,12,13]. In this article, we propose the analysis of facial micro expressions as a possible solution to these problems. Micro facial expressions are facial expressions that occur in a fraction of a second. They can provide accurate information about a person's actual emotional state, regardless of culture, language, and personal background. This information can, therefore, be exploited to create intelligent user interfaces, capable of capturing the real emotions of large communities of individuals, thus promoting cultural and social inclusion among individuals coming from different realities and belonging to different categories, including disadvantaged and at-risk groups, as well as vulnerable people. There are various applications and scenarios in which such intelligent interfaces could provide significant benefits, including recommender systems [14,15,16], intelligent tutoring systems [17], and, more generally, smart cities [18]. To demonstrate the feasibility of our idea, we report the preliminary results of a user study conducted by recording the micro facial expressions of some testers in response to certain perceptual stimuli. Although this study was carried out in a specific domain (i.e., cultural heritage [19,20]) and on a very limited and skewed sample of users, the results obtained show the existence of correlations between some action units (i.e., single components of the muscular movement in which facial expressions can be broken down) and emotional reactions. They also show that it is possible to identify common correlations within different categories of individuals. This somehow confirms our initial idea and encourages us to continue our experimental analysis, extending it to a more significant and heterogeneous sample of users.
Kinesics is the science that studies body language. According to the anthropologist Ray Birdwhistell, who coined this term in 1952, this science allows us to interpret a person's thoughts, feelings, and emotions by analyzing her facial expressions, gestures, posture, gaze, and movements of the legs and arms [21]. Birdwhistell's theories were highly regarded over the years and it is well known that mere verbal communication represents only a small part of the message that allows two individuals to convey information to each other. According to the 7-38-55 Rule developed by Albert Mehrabian in the 1970s [22], communication takes place in three ways: the content (what is communicated), tone (how it is communicated), and body language (posture, expressions, etc). The digits that appear in the rule name indicate the percentage of the relevance of these ways: 7% the content of the message, 38% the tone of the voice, 55% the body language.
The kinesic system of signification and signaling includes the movements of the body, face, and eyes [23]. Facial expressions manifest the intentions of the subject based on the context and depending on this there are facial expressions that differ substantially, also giving the listener the possibility to understand the state of mind of her interlocutor. In 1979 Paul Ekman and Wallace V. Friesen, based on the previously developed study by Swedish anatomist Carl-Herman Hjortsjö [24], proposed the Facial Action Coding System (FACS) [23], an anatomically accurate system to describe all visually distinguishable facial movements.
The FACS decoding system explores facial expressions by breaking them down into the smallest fundamental units, the action units (AUs), giving each one a meaning. Ekman and Friesen cataloged 44 AUs describing changes in facial expressions and 14 AUs mapping changes in the eye gaze direction and the head orientation. The AUs play a fundamental role in the recognition of emotions, movements, and attitudes, not only of the face but also of the body, allowing us to analyze the state of mind of the subject. The combination of the AUs enables us to map the four main emotions, namely, happiness, sadness, anger, and fear [25].
The research questions underlying the experimental analysis we performed are the following: is there a correlation between the micro facial expressions of an observer and her degree of appreciation (i.e., rating) of an artwork? Is it possible to identify correlations shared by specific categories of users? To answer these questions, it was necessary to collect the data that could allow us to verify our initial assumptions.
At the beginning of our research activity, we had planned real experimentation in a suitable place to verify our hypotheses, for example, a museum. Unfortunately, the limitations imposed by the COVID-19 pandemic did not allow us to follow this road. Consequently, to collect data it was necessary to develop an online application. First of all, we developed a website1 that had mainly two functions. The first function was to simulate a visit sharing the same characteristics as a visit to a real museum. For this purpose, we selected some artworks from those exhibited at the National Gallery of Modern and Contemporary Art2 in Rome, Italy. The selection was made in such a way as to be able to show the user works as different as possible. The second function was to collect information about the visitor. In particular, we were interested in acquiring data relating to her demographic profile, degree of appreciation of the work displayed at that time, and resulting micro facial expressions. Specifically, participants were shown eight artworks and asked to rate each of them on a five-point Likert scale. Meanwhile, the participants were recorded through the webcam of their device while viewing each artwork. Demographic information was collected through a final questionnaire. Specifically, the demographic data relating to the users who participated in the experimental trials are shown in Table 1. The participants were 73, almost equally distributed between females and males, and aged mostly between 21 and 29. Most participants had a high school diploma and were mainly university students. Once the dataset was collected, it was necessary to process the recorded videos using facial recognition software. We employed two different software tools for this purpose: OpenFace3 , an opensource toolkit capable of performing action unit analysis, and iMotions4 , a proprietary software.
Let us now analyze the results returned by the two analysis software. Table 2 shows the average values, standard deviations, as well as the minimum and maximum values, calculated on the whole dataset. First of all, we can observe that the iMotions software returns more information than OpenFace and that the two software tools sometimes analyze the same micro expressions. The mean of the individual action units is often less than the standard deviation. At the same time, the minimum values differ highly from the maximum values. These results, therefore, indicate the tendency of visitors to assume a neutral expression for most of the time except in rare moments. The attention score, namely, the attention showed by the visitor while observing the artwork, is noteworthy. The average value is very close to the maximum value, and the deviation is very low. We can, hence, conclude that most testers kept high their level of attention during the virtual visit. Table 3 shows the value of Spearman's correlation coefficient of the ratings assigned by the testers to the individual works and the average score obtained by the features for each video. We can immediately notice a high correlation value between ratings and eye closure. The same thing happens for perceived sadness. The negative value of these correlations indicates that a high value of the feature corresponds to a low rating attributed to the work. We then verified if there were any correlations shared by some categories of testers. More specifically, we grouped the data based on gender, the rating attributed to the artwork, and the number of recognized artworks. Table 4 reports the values returned by OpenFace. We note a positive correlation value between the rating and the cheek raise action unit related to women. The same thing happens for the distension of the eyelids, both for women and for those who have recognized few works. Finally, for those who assigned a low rating, we found a negative correlation for the lifting of the lips and their sinking. In Table 5, we can instead observe the correlation values calculated on the results of iMotions. We can observe how eye closure is negatively correlated in the two groups. Also, sadness is negatively correlated in the two groups. Joy is also negatively correlated. And, finally, fear and contempt are negatively correlated for the same group of people who recognized a few works. These results, therefore, show how all features are negatively correlated, thus expressing that a high score of a given variable corresponds to a lower score. Further analyzes that we cannot report for reasons of space show how visitors who did not like the works expressed their low ratings more clearly.
The ultimate goal of our research activities was to verify whether facial micro expressions can be exploited to create interfaces that can adapt differently depending on the characteristics of the active user. If so, it would be possible to foster cultural and social inclusion between individuals from different backgrounds and belonging to different categories, including disadvantaged and at-risk categories as well as vulnerable people. In particular, from the experimental results, it emerged how it is pos-sible to identify some correlation between facial micro expressions and the degree of appreciation of an object, specifically an artwork. It is also possible to identify correlations within some homogeneous groups of testers.
Our experimental analysis is very simplified and also suffers from numerous limitations. Among others, it is evident as follows:
• it was performed in a specific domain, namely that of cultural heritage; • the micro facial expressions were collected in response to a specific stimulus, that is, the vision of an artwork; • the data was collected through a virtual and not live experimentation; • the sample of users was very limited; • the sample of users was mostly made up of university students, so it was anything but heterogeneous.
A much more extensive and rigorous experimental analysis is therefore needed, including further categories of users, scenarios (e.g., [26,27,28]), and information (e.g., [29]). Only in this way we could indeed draw definitive conclusions on the existence of correlations between micro facial expressions and categories of testers.