Exploiting Micro Facial Expressions for More Inclusive
User Interfaces
Alessio Ferratoa , Carla Limongellia , Mauro Mezzinib and Giuseppe Sansonettia
a
Department of Engineering, Roma Tre University, Via della Vasca Navale 79, 00146 Rome, Italy
b
Department of Education, Roma Tre University, Viale del Castro Pretorio 20, 00185 Rome, Italy
Abstract
Current image/video acquisition and analysis techniques allow for not only the identification and classification of objects
in a scene but also more sophisticated processing. For example, there are video cameras today able to capture micro facial
expressions, namely, facial expressions that occur in a fraction of a second. Such micro expressions can provide useful
information to define a person’s emotional state. In this article, we propose to use these features to collect useful information
for designing and implementing increasingly effective interactive technologies. In particular, facial micro expressions could
be used to develop interfaces capable of fostering the social and cultural inclusion of users belonging to different realities and
categories. The preliminary experimental results obtained by recording the reactions of individuals while observing artworks
demonstrate the existence of correlations between the action units (i.e., single components of the muscular movement in
which it is possible to break down facial expressions) and the emotional reactions of a sample of users, as well as correlations
within some homogeneous groups of testers.
Keywords
User interfaces, User modeling, Emotion recognition, Computer vision
1. Introduction and Background state, regardless of culture, language, and personal back-
ground. This information can, therefore, be exploited to
Systems capable of identifying a user’s emotional state create intelligent user interfaces, capable of capturing
starting from her behavior are becoming more and more the real emotions of large communities of individuals,
popular [1]. Among these, Automatic Facial Expression thus promoting cultural and social inclusion among indi-
Analysis (AFEA) [2] systems are of particular importance. viduals coming from different realities and belonging to
Facial expressions can be defined as facial changes in re- different categories, including disadvantaged and at-risk
sponse to a person’s internal emotional states, intentions, groups, as well as vulnerable people. There are vari-
or social communications [3]. This research topic is cer- ous applications and scenarios in which such intelligent
tainly not new if we consider that Darwin in 1872 had interfaces could provide significant benefits, including
already addressed the subject in [4]. Since then, there recommender systems [14, 15, 16], intelligent tutoring
have been several attempts by behavioral scientists to systems [17], and, more generally, smart cities [18]. To
conceive methods and models for the automatic analysis demonstrate the feasibility of our idea, we report the
of facial expressions on image sequences [5, 6]. These preliminary results of a user study conducted by record-
studies have laid the foundations for the realization of ing the micro facial expressions of some testers in re-
computer systems able to help us understand this natu- sponse to certain perceptual stimuli. Although this study
ral form of communication among human beings (e.g., was carried out in a specific domain (i.e., cultural her-
see [7, 8, 9, 10]). Such systems, although very efficient, itage [19, 20]) and on a very limited and skewed sample of
are inevitably affected by context, culture, genre and so users, the results obtained show the existence of correla-
on [11, 12, 13]. In this article, we propose the analysis of tions between some action units (i.e., single components
facial micro expressions as a possible solution to these of the muscular movement in which facial expressions
problems. Micro facial expressions are facial expressions can be broken down) and emotional reactions. They also
that occur in a fraction of a second. They can provide show that it is possible to identify common correlations
accurate information about a person’s actual emotional within different categories of individuals. This somehow
confirms our initial idea and encourages us to continue
Joint Proceedings of the ACM IUI 2021 Workshops, April 13-17, 2021, our experimental analysis, extending it to a more signifi-
College Station, USA
cant and heterogeneous sample of users.
Envelope-Open ale.ferrato@stud.uniroma3.it (A. Ferrato);
limongel@dia.uniroma3.it (C. Limongelli);
mauro.mezzini@uniroma3.it (M. Mezzini);
gsansone@dia.uniroma3.it (G. Sansonetti) 2. Kinesics
Orcid 0000-0003-4953-1390 (G. Sansonetti)
© 2021 Copyright © 2021 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0).
Kinesics is the science that studies body language. Ac-
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org) cording to the anthropologist Ray Birdwhistell, who coined
this term in 1952, this science allows us to interpret a necessary to collect the data that could allow us to verify
person’s thoughts, feelings, and emotions by analyzing our initial assumptions.
her facial expressions, gestures, posture, gaze, and move-
ments of the legs and arms [21]. Birdwhistell’s theories 3.1. The development of a data collection
were highly regarded over the years and it is well known
that mere verbal communication represents only a small
system
part of the message that allows two individuals to convey At the beginning of our research activity, we had planned
information to each other. According to the 7-38-55 Rule real experimentation in a suitable place to verify our hy-
developed by Albert Mehrabian in the 1970s [22], com- potheses, for example, a museum. Unfortunately, the
munication takes place in three ways: the content (what limitations imposed by the COVID-19 pandemic did not
is communicated), tone (how it is communicated), and allow us to follow this road. Consequently, to collect data
body language (posture, expressions, etc). The digits that it was necessary to develop an online application. First
appear in the rule name indicate the percentage of the of all, we developed a website1 that had mainly two func-
relevance of these ways: 7% the content of the message, tions. The first function was to simulate a visit sharing
38% the tone of the voice, 55% the body language. the same characteristics as a visit to a real museum. For
this purpose, we selected some artworks from those ex-
2.1. Facial expressions (FACS) hibited at the National Gallery of Modern and Contempo-
rary Art2 in Rome, Italy. The selection was made in such
The kinesic system of signification and signaling includes a way as to be able to show the user works as different as
the movements of the body, face, and eyes [23]. Facial possible. The second function was to collect information
expressions manifest the intentions of the subject based about the visitor. In particular, we were interested in
on the context and depending on this there are facial ex- acquiring data relating to her demographic profile, de-
pressions that differ substantially, also giving the listener gree of appreciation of the work displayed at that time,
the possibility to understand the state of mind of her and resulting micro facial expressions. Specifically, par-
interlocutor. In 1979 Paul Ekman and Wallace V. Friesen, ticipants were shown eight artworks and asked to rate
based on the previously developed study by Swedish each of them on a five-point Likert scale. Meanwhile,
anatomist Carl-Herman Hjortsjö [24], proposed the Fa- the participants were recorded through the webcam of
cial Action Coding System (FACS) [23], an anatomically their device while viewing each artwork. Demographic
accurate system to describe all visually distinguishable information was collected through a final questionnaire.
facial movements. Specifically, the demographic data relating to the users
who participated in the experimental trials are shown
2.1.1. Action Units (AUs) in Table 1. The participants were 73, almost equally dis-
tributed between females and males, and aged mostly
The FACS decoding system explores facial expressions
between 21 and 29. Most participants had a high school
by breaking them down into the smallest fundamental
diploma and were mainly university students. Once the
units, the action units (AUs), giving each one a meaning.
dataset was collected, it was necessary to process the
Ekman and Friesen cataloged 44 AUs describing changes
recorded videos using facial recognition software. We
in facial expressions and 14 AUs mapping changes in the
employed two different software tools for this purpose:
eye gaze direction and the head orientation. The AUs
OpenFace3 , an opensource toolkit capable of perform-
play a fundamental role in the recognition of emotions,
ing action unit analysis, and iMotions4 , a proprietary
movements, and attitudes, not only of the face but also
software.
of the body, allowing us to analyze the state of mind of
the subject. The combination of the AUs enables us to
map the four main emotions, namely, happiness, sadness, 4. Data Analysis
anger, and fear [25].
Let us now analyze the results returned by the two analy-
sis software. Table 2 shows the average values, standard
3. Data Collection deviations, as well as the minimum and maximum val-
ues, calculated on the whole dataset. First of all, we can
The research questions underlying the experimental anal-
observe that the iMotions software returns more infor-
ysis we performed are the following: is there a correlation
mation than OpenFace and that the two software tools
between the micro facial expressions of an observer and
her degree of appreciation (i.e., rating) of an artwork? 1
https://www.raccoltadati.tk/
Is it possible to identify correlations shared by specific 2
https://lagallerianazionale.com/en/
categories of users? To answer these questions, it was 3
https://github.com/TadasBaltrusaitis/OpenFace
4
https://imotions.com/
Table 1 maximum value, and the deviation is very low. We can,
Demographics of the 73 users involved in the experimental hence, conclude that most testers kept high their level
analysis of attention during the virtual visit. Table 3 shows the
value of Spearman’s correlation coefficient of the ratings
Item Frequency assigned by the testers to the individual works and the
Female 37 average score obtained by the features for each video. We
Gender
Male 36
Under 18 3 Table 3
18-20 5 Spearman’s correlation coefficient
21-29 40
Age 30-39 3
40-49 3 iMotions OpenFace
50-59 12 AU & Emotions Spearman’s Index
Over 60 7 Inner Brow Raise -0.07 -0.06
Primary school 1 Outer Brow Raise -0.01 -0.05
8th grade diploma 9 Brow Lower -0.05 -0.06
Education High school diploma 41 Upper Lid Raise -0.05
University degree 19 Cheek Raise 0.00 -0.05
PhD 3 Lid Tighten -0.05 0.06
Unemployed 3 Nose Wrinkle -0.04 -0,04
Student 39 Upper Lip Raise -0.03 -0.04
Public employee 7 Lip Corner Puller 0.00
Profession Dimpler -0.02 -0.03
Private employee 14
Self-employed 7 Lip Corner Depressor -0.04 -0.06
Retired 3 Chin Raise 0.01 -0.07
Lip Stretch -0.09 -0.04
Lid Tighten -0.08
Mouth Open -0.01 0.00
Table 2 Jaw Drop -0.05 -0.02
Summary table of the output from the two software tools Blink -0.08
iMotions OpenFace Lip Suck -0.03
AU & Emotions Mean Std Min Max Mean Std Min Max
Inner Brow Raise 5,099658 12,94021 0 80,29622 0,168434 0,141843 0,039858 1,462658 Lip Press -0.05
Brow Raise 3,565345 8,847247 0 55,49171 0,085252 0,061114 0,021103 0,478671
Brow Lower 5,334099 12,40427 0 76,77342 0,765825 0,739402 0 3,596304 Lip Pucker -0.06
Upper Lid Raiser 0,055565 0,031256 0,014244 0,245095
Cheek Raise 3,659209 10,67665 0 69,96562 0,390288 0,466435 0 2,387549 Eye Closure -0.17**
Lid Tighten 0,93787 2,604525 0 23,44269 0,616453 0,719307 0 3,199208
Nose Wrinkle 0,973915 3,62885 0 44,94059 0,063658 0,054983 0,013989 0,350426 Eye Widen 0.03
Upper Lip Raise 1,135299 4,613869 0 44,57584 0,555492 0,527603 0 3,205763
Lip Corner Puller 0,397487 0,473547 0 2,572438 Smile -0.01
Dimpler 3,837253 7,598816 0 54,32411 0,570261 0,564813 0 2,724876
Lip Corner Depressor 1,766322 4,586096 0 41,22998 0,189903 0,220943 0,036511 1,946785 Smirk 0.04
Chin Raise 2,785176 5,867499 0 37,18328 0,407547 0,2544 0,080133 1,586465
Lip Stretch 2,535029 7,484421 0 61,21821 0,117077 0,11238 0,030426 1,131618 Engagement -0.04
Lip Tighten 0,93787 2,604525 0 23,44269 0,121904 0,123215 0,018549 0,929964
Mouth Open 6,867683 11,51858 0 66,08074 0,365305 0,331243 0,064533 2,580889 Attention -0.05
Jaw Drop 3,772275 6,797671 0 42,74697 0,36226 0,30048 0,0674 2,31789
Blink 0,169887 0,066811 0,041817 0,383651 Anger -0.05
Lip Suck 5,259716 9,547491 0 58,75693
Lip Press 2,926959 5,577136 0 31,28165 Sadness -0.13*
Lip Pucker 2,870787 7,183146 0 46,96508
Eye Closure 1,966987 3,09202 0 30,51927 Disgust -0.02
Eye Widen 3,038526 7,873084 0 62,36883
Smile 7,651248 16,54695 0 82,14044 Joy -0.09
Smirk 2,030771 5,974433 0 62,60415
Engagement 15,29063 20,46839 0 88,82519 Surprise -0.07
Attention 93,17853 11,72724 15,89159 98,63756
Anger 0,473087 1,830745 0 21,59573 Fear -0.05
Sadness 0,869082 2,900364 0 28,76604
Disgust 1,297257 4,502729 0 61,42045 Contempt -0.07
Joy 5,829057 15,26311 0 83,61379
Surprise 1,364944 3,271783 0 31,10703
Fear 0,468503 1,842737 0 16,90147
Contempt 1,431101 5,146328 0 64,36057
can immediately notice a high correlation value between
ratings and eye closure. The same thing happens for per-
sometimes analyze the same micro expressions. The ceived sadness. The negative value of these correlations
mean of the individual action units is often less than indicates that a high value of the feature corresponds to
the standard deviation. At the same time, the minimum a low rating attributed to the work. We then verified if
values differ highly from the maximum values. These there were any correlations shared by some categories of
results, therefore, indicate the tendency of visitors to as- testers. More specifically, we grouped the data based on
sume a neutral expression for most of the time except gender, the rating attributed to the artwork, and the num-
in rare moments. The attention score, namely, the atten- ber of recognized artworks. Table 4 reports the values
tion showed by the visitor while observing the artwork, returned by OpenFace. We note a positive correlation
is noteworthy. The average value is very close to the value between the rating and the cheek raise action unit
Table 4 sible to identify some correlation between facial micro
Correlations between homogeneous groups in OpenFace expressions and the degree of appreciation of an object,
Groups Male Female
Low
ratings
High
ratings
Low
frequency
High
frequency
Few
recognized
Many
recognized
Low
interest
High
interest
specifically an artwork. It is also possible to identify cor-
# Measurements 24 22 125 165 19 5 41 1 0 15
Inner Brow Raise
Brow Raise
-0.11
-0.05
-0.01
-0.04
-0.08
-0.08
0.01
0.06
-0.01
0.05
0.07
0.03
-0.06
-0.05
0.39
0.39
0
0
-0.07
-0.04
relations within some homogeneous groups of testers.
Brow Lower -0.02 -0.09 -0.03 0.01 -0.06 0.00 -0.06 -0.39 0 -0.15
Upper Lid Raiser
Cheek Raise
Lid Tighten
-0.11
-0.04
-0.05
0.02
0.15*
0.17*
-0.11
-0.06
-0.05
-0.07
-0.04
0.01
0.05
0.05
0.04
-0.06
0.06
0.19
-0.06
0.06
0.12*
0.23
-0.49
0.71*
0
0
0
-0.04
-0.02
0.00
Our experimental analysis is very simplified and also
Nose Wrinkle
Upper Lip Raise
Lip Corner Puller
-0.06
-0.02
-0.05
-0.02
-0.09
0.05
-0.15
-0.19*
-0.19*
-0.06
-0.07
-0.01
-0.08
0.06
0.06
0.38*
-0.16
-0.03
-0.05
-0.06
0.00
0.00
-0.05
0.10
0
0
0
0.02
-0.03
-0.01
suffers from numerous limitations. Among others, it is
Dimpler
Lip Corner Depressor
Chin Raise
-0.02
-0.10
-0.10
-0.03
0.00
-0.03
-0.17
-0.05
-0.13
-0.07
0.04
-0.05
0.10
0.01
-0.08
-0.06
0.18
0.20
-0.04
-0.06
-0.06
0.15
-0.28
-0.23
0
0
0
-0.09
-0.10
-0.09
evident as follows:
Lip Stretch -0.09 0.03 0.00 0.00 0.06 0.10 -0.05 -0.15 0 -0.08
Lip Tighten -0.08 -0.05 -0.09 -0.01 -0.07 0.04 -0.10 -0.54 0 -0.04
Mouth Open
Jaw Drop
Blink
-0.04
-0.06
-0.08
0.04
0.04
-0.08
-0.14
-0.03
-0.06
0.07
0.01
-0.09
0.03
0.03
-0.05
0.18
0.16
0.33*
0.00
-0.02
-0.08
0.13
-0.31
-0.28
0
0
0
-0.05
-0.05
-0.10
• it was performed in a specific domain, namely
p < .0001 ‘****’; p < .001 ‘***’, p < .01 ‘**’, p < .05 ‘*’
that of cultural heritage;
• the micro facial expressions were collected in re-
sponse to a specific stimulus, that is, the vision
related to women. The same thing happens for the dis-
of an artwork;
tension of the eyelids, both for women and for those who
• the data was collected through a virtual and not
have recognized few works. Finally, for those who as-
live experimentation;
signed a low rating, we found a negative correlation for
• the sample of users was very limited;
the lifting of the lips and their sinking. In Table 5, we
• the sample of users was mostly made up of uni-
can instead observe the correlation values calculated on
versity students, so it was anything but heteroge-
the results of iMotions. We can observe how eye closure
neous.
Table 5
A much more extensive and rigorous experimental anal-
Correlations between homogeneous groups in iMotions ysis is therefore needed, including further categories
Groups Male Female
Low High Low High Few Many Low High
of users, scenarios (e.g., [26, 27, 28]), and information
ratings ratings frequency frequency recognized recognized interest interest
# Measurements
Brow Furrow
24
-0.04
22
-0.06
125
-0.05
165
0.04
19
0.00
5
0.11
41
-0.05
1
0.28
0
0
15
-0.08
(e.g., [29]). Only in this way we could indeed draw defini-
tive conclusions on the existence of correlations between
Brow Raise -0.02 0.00 -0.08 0.00 -0.06 0.12 -0.01 0.33 0 0.03
Engagement -0.12 0.04 -0.12 0.04 -0.03 -0.04 -0.04 0.00 0 -0.11
Lip Corner Depressor -0.07 0.01 0.05 0.03 -0.05 0.35* -0.06 -0.49 0 0.02
Smile -0.13 0.11 -0.19* 0.01 0.05 -0.07 0.04 -0.18 0 -0.11
Attention
Inner Brow Raise
Eye Closure
0.00
-0.13
-0.13
-0.10
0.01
-0.21**
0.15
0.06
-0.02
-0.15
-0.04
-0.07
-0.22**
-0.09
-0.20*
-0.14
0.17
0.19
-0.03
-0.06
-0.21***
-0.69
-0.67
-0.08
0
0
0
-0.03
-0.04
-0.09
micro facial expressions and categories of testers.
Nose Wrinkle -0.06 0.01 -0.14 -0.03 -0.02 0.01 -0.01 0.08 0 -0.07
Upper Lip Raise -0.05 0.01 -0.12 -0.05 -0.01 0.08 -0.01 0.31 0 -0.04
Lip Suck -0.07 0.00 -0.09 0.02 -0.05 -0.02 -0.02 0.80* 0 -0.08
Lip Press -0.09 0.00 -0.09 -0.02 -0.04 0.03 -0.05 0.44 0 -0.06
References
Mouth Open -0.05 0.03 -0.11 0.02 0.08 0.14 0.01 0.10 0 -0.11
Chin Raise -0.06 0.11 -0.08 0.06 -0.06 -0.04 0.00 0.28 0 0.02
Smirk -0.01 0.12 -0.13 0.06 0.06 -0.11 0.02 0.69 0 0.06
Lip Pucker 0.06 0.06 -0.13 0.04 0.03 -0.06 0.02 -0.05 0 0.07
Anger -0.05 -0.04 0.02 0.06 -0.01 0.16 -0.08 0.33 0 -0.08
Sadness -0.14* -0.13 -0.04 0.00 -0.19* 0.13 -0.15** -0.23 0 -0.06
Disgust 0.01 -0.03 -0.04 0.02 0.00 0.33* -0.05 -0.10 0 0.05
[1] X. Alameda-Pineda, E. Ricci, N. Sebe, Multimodal
Joy -0.16* 0.00 -0.16 -0.02 -0.05 0.01 -0.05 -0.36 0 -0.17*
Surprise -0.07 -0.07 -0.12 -0.02 -0.11 -0.21 -0.08 0.33 0 -0.01
Fear -0.05 -0.05 0.02 -0.07 -0.02 0.06 -0.12* -0.08 0 -0.05
Contempt -0.07 -0.06 0.02 0.06 -0.08 0.04 -0.12* 0.72* 0 0.01
Cheek Raise
Dimpler
Eye Widen
-0.11
-0.09
0.04
0.11
0.05
0.03
-0.17
-0.12
-0.02
0.03
-0.04
-0.08
0.05
0.02
0.13
-0.05
-0.04
-0.35*
0.04
-0.02
-0.04
0.03
0.41
-0.08
0
0
0
-0.10
-0.07
0.00
behavior analysis in the wild: An introduction, in:
Lid Tighten
Lip Stretch
-0.13
-0.11
0.02
-0.07
-0.07
-0.16
p < .0001 ‘****’; p < .001 ‘***’, p < .01 ‘**’, p < .05 ‘*’
-0.02
-0.12
-0.18*
-0.03
0.27
-0.25
-0.01
-0.07
-0.39
-0.08
0
0
-0.04
-0.12 X. Alameda-Pineda, E. Ricci, N. Sebe (Eds.), Mul-
timodal Behavior Analysis in the Wild, Computer
is negatively correlated in the two groups. Also, sadness Vision and Pattern Recognition, Academic Press,
is negatively correlated in the two groups. Joy is also 2019, pp. 1–8.
negatively correlated. And, finally, fear and contempt are [2] B. T. Hung, L. M. Tien, Facial expression recognition
negatively correlated for the same group of people who with cnn-lstm, in: R. Kumar, N. H. Quang, V. Ku-
recognized a few works. These results, therefore, show mar Solanki, M. Cardona, P. K. Pattnaik (Eds.), Re-
how all features are negatively correlated, thus express- search in Intelligent and Computing in Engineering,
ing that a high score of a given variable corresponds to a Springer Singapore, Singapore, 2021, pp. 549–560.
lower score. Further analyzes that we cannot report for [3] Y. Tian, T. Kanade, J. F. Cohn, Facial expression
reasons of space show how visitors who did not like the recognition, in: S. Z. Li, A. K. Jain (Eds.), Hand-
works expressed their low ratings more clearly. book of Face Recognition, Springer London, Lon-
don, 2011, pp. 487–519.
[4] C. Darwin, The expression of the emotions in man
5. Conclusions and Future Works and animals, London, UK: John Marry (1872).
[5] E. Friesen, P. Ekman, The facial action coding sys-
The ultimate goal of our research activities was to ver- tem: a technique for the measurement of facial
ify whether facial micro expressions can be exploited to movement, Palo Alto 3 (1978).
create interfaces that can adapt differently depending on [6] P. Ekman, K. R. Scherer, Handbook of methods in
the characteristics of the active user. If so, it would be nonverbal behavior research, Cambridge University
possible to foster cultural and social inclusion between Press, 1982.
individuals from different backgrounds and belonging to [7] M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek,
different categories, including disadvantaged and at-risk I. Fasel, J. Movellan, Automatic recognition of fa-
categories as well as vulnerable people. In particular, cial actions in spontaneous expressions, Journal of
from the experimental results, it emerged how it is pos- Multimedia 1 (2006) 22–35.
[8] H. Gunes, M. Piccardi, Automatic temporal segment dentlitteratur, 1969.
detection and affect recognition from face and body [25] C. G. Kohler, T. Turner, N. M. Stolar, W. B. Bilker,
display, IEEE Transactions on Systems, Man, and C. M. Brensinger, R. E. Gur, R. C. Gur, Differences
Cybernetics, Part B (Cybernetics) 39 (2008) 64–84. in facial expressions of four universal emotions,
[9] J. Lien, T. Kanade, J. Cohn, C. Li, Detection, tracking, Psychiatry Research 128 (2004) 235 – 244.
and classification of action units in facial expression, [26] S. Caldarelli, D. F. Gurini, A. Micarelli, G. Sansonetti,
Robotics and Autonomous Systems 31 (2000). A signal-based approach to news recommendation,
[10] Y.-l. Tian, T. Kanade, J. F. Cohn, Recognizing action in: CEUR Workshop Proceedings, volume 1618,
units for facial expression analysis, IEEE Trans. CEUR-WS.org, Aachen, Germany, 2016.
Pattern Anal. Mach. Intell. 23 (2001) 97–115. [27] M. Onori, A. Micarelli, G. Sansonetti, A comparative
[11] J. Carroll, J. Russell, Do facial expressions signal analysis of personality-based music recommender
specific emotions? judging emotion from the face systems, in: CEUR Workshop Proceedings, volume
in context., Journal of personality and social psy- 1680, CEUR-WS.org, Aachen, Germany, 2016.
chology 70 2 (1996) 205–18. [28] D. Valeriani, G. Sansonetti, A. Micarelli, A compar-
[12] J. Russell, Culture and the categorization of emo- ative analysis of state-of-the-art recommendation
tions., Psychological bulletin 110 3 (1991) 426–50. techniques in the movie domain, Lecture Notes in
[13] J. Russell, Is there universal recognition of emo- Computer Science 12252 LNCS (2020) 104–118.
tion from facial expression? a review of the cross- [29] M. Saneiro, O. Santos, S. Salmeron-Majadas, J. Bot-
cultural studies., Psychological bulletin 115 1 (1994). icario, Towards emotion detection in educational
[14] G. Sansonetti, Point of interest recommendation scenarios from facial expressions and body move-
based on social and linked open data, Personal and ments through multimodal approaches, The Scien-
Ubiquitous Computing 23 (2019) 199–214. tific World Journal (2014).
[15] H. A. M. Hassan, G. Sansonetti, F. Gasparetti, A. Mi-
carelli, Semantic-based tag recommendation in sci-
entific bookmarking systems, in: Proceedings of
the 12th ACM Conference on Recommender Sys-
tems, ACM, New York, NY, USA, 2018, pp. 465–469.
[16] A. Fogli, G. Sansonetti, Exploiting semantics for
context-aware itinerary recommendation, Personal
and Ubiquitous Computing 23 (2019) 215–231.
[17] M. Chang, G. D’Aniello, M. Gaeta, F. Orciuoli,
D. Sampson, C. Simonelli, Building ontology-driven
tutoring models for intelligent tutoring systems us-
ing data mining, IEEE Access 8 (2020) 48151–48162.
[18] G. D’Aniello, M. Gaeta, F. Orciuoli, G. Sansonetti,
F. Sorgente, Knowledge-based smart city service
system, Electronics 9 (2020).
[19] G. Sansonetti, F. Gasparetti, A. Micarelli, F. Cena,
C. Gena, Enhancing cultural recommendations
through social and linked open data, User Modeling
and User-Adapted Interaction 29 (2019) 121–159.
[20] M. Mezzini, C. Limongelli, G. Sansonetti,
C. De Medio, Tracking museum visitors through
convolutional object detectors, in: Adjunct
Publication of UMAP ’20, ACM, New York, NY,
USA, 2020, p. 352–355.
[21] R. L. Birdwhistell, Kinesics and context: Essays on
body motion communication, University of Penn-
sylvania press, 2010.
[22] A. Mehrabian, M. Wiener, Decoding of inconsistent
communications, Journal of personality and social
psychology 6 (1967) 109—114.
[23] P. Ekman, W. Friesen, Facial Action Coding System,
Consulting Psychologists Press, 1978.
[24] C. Hjortsjö, Man’s Face and Mimic Language, Stu-