Exploiting Micro Facial Expressions for More Inclusive User Interfaces Alessio Ferratoa , Carla Limongellia , Mauro Mezzinib and Giuseppe Sansonettia a Department of Engineering, Roma Tre University, Via della Vasca Navale 79, 00146 Rome, Italy b Department of Education, Roma Tre University, Viale del Castro Pretorio 20, 00185 Rome, Italy Abstract Current image/video acquisition and analysis techniques allow for not only the identification and classification of objects in a scene but also more sophisticated processing. For example, there are video cameras today able to capture micro facial expressions, namely, facial expressions that occur in a fraction of a second. Such micro expressions can provide useful information to define a person’s emotional state. In this article, we propose to use these features to collect useful information for designing and implementing increasingly effective interactive technologies. In particular, facial micro expressions could be used to develop interfaces capable of fostering the social and cultural inclusion of users belonging to different realities and categories. The preliminary experimental results obtained by recording the reactions of individuals while observing artworks demonstrate the existence of correlations between the action units (i.e., single components of the muscular movement in which it is possible to break down facial expressions) and the emotional reactions of a sample of users, as well as correlations within some homogeneous groups of testers. Keywords User interfaces, User modeling, Emotion recognition, Computer vision 1. Introduction and Background state, regardless of culture, language, and personal back- ground. This information can, therefore, be exploited to Systems capable of identifying a user’s emotional state create intelligent user interfaces, capable of capturing starting from her behavior are becoming more and more the real emotions of large communities of individuals, popular [1]. Among these, Automatic Facial Expression thus promoting cultural and social inclusion among indi- Analysis (AFEA) [2] systems are of particular importance. viduals coming from different realities and belonging to Facial expressions can be defined as facial changes in re- different categories, including disadvantaged and at-risk sponse to a person’s internal emotional states, intentions, groups, as well as vulnerable people. There are vari- or social communications [3]. This research topic is cer- ous applications and scenarios in which such intelligent tainly not new if we consider that Darwin in 1872 had interfaces could provide significant benefits, including already addressed the subject in [4]. Since then, there recommender systems [14, 15, 16], intelligent tutoring have been several attempts by behavioral scientists to systems [17], and, more generally, smart cities [18]. To conceive methods and models for the automatic analysis demonstrate the feasibility of our idea, we report the of facial expressions on image sequences [5, 6]. These preliminary results of a user study conducted by record- studies have laid the foundations for the realization of ing the micro facial expressions of some testers in re- computer systems able to help us understand this natu- sponse to certain perceptual stimuli. Although this study ral form of communication among human beings (e.g., was carried out in a specific domain (i.e., cultural her- see [7, 8, 9, 10]). Such systems, although very efficient, itage [19, 20]) and on a very limited and skewed sample of are inevitably affected by context, culture, genre and so users, the results obtained show the existence of correla- on [11, 12, 13]. In this article, we propose the analysis of tions between some action units (i.e., single components facial micro expressions as a possible solution to these of the muscular movement in which facial expressions problems. Micro facial expressions are facial expressions can be broken down) and emotional reactions. They also that occur in a fraction of a second. They can provide show that it is possible to identify common correlations accurate information about a person’s actual emotional within different categories of individuals. This somehow confirms our initial idea and encourages us to continue Joint Proceedings of the ACM IUI 2021 Workshops, April 13-17, 2021, our experimental analysis, extending it to a more signifi- College Station, USA cant and heterogeneous sample of users. Envelope-Open ale.ferrato@stud.uniroma3.it (A. Ferrato); limongel@dia.uniroma3.it (C. Limongelli); mauro.mezzini@uniroma3.it (M. Mezzini); gsansone@dia.uniroma3.it (G. Sansonetti) 2. Kinesics Orcid 0000-0003-4953-1390 (G. Sansonetti) © 2021 Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Kinesics is the science that studies body language. Ac- CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) cording to the anthropologist Ray Birdwhistell, who coined this term in 1952, this science allows us to interpret a necessary to collect the data that could allow us to verify person’s thoughts, feelings, and emotions by analyzing our initial assumptions. her facial expressions, gestures, posture, gaze, and move- ments of the legs and arms [21]. Birdwhistell’s theories 3.1. The development of a data collection were highly regarded over the years and it is well known that mere verbal communication represents only a small system part of the message that allows two individuals to convey At the beginning of our research activity, we had planned information to each other. According to the 7-38-55 Rule real experimentation in a suitable place to verify our hy- developed by Albert Mehrabian in the 1970s [22], com- potheses, for example, a museum. Unfortunately, the munication takes place in three ways: the content (what limitations imposed by the COVID-19 pandemic did not is communicated), tone (how it is communicated), and allow us to follow this road. Consequently, to collect data body language (posture, expressions, etc). The digits that it was necessary to develop an online application. First appear in the rule name indicate the percentage of the of all, we developed a website1 that had mainly two func- relevance of these ways: 7% the content of the message, tions. The first function was to simulate a visit sharing 38% the tone of the voice, 55% the body language. the same characteristics as a visit to a real museum. For this purpose, we selected some artworks from those ex- 2.1. Facial expressions (FACS) hibited at the National Gallery of Modern and Contempo- rary Art2 in Rome, Italy. The selection was made in such The kinesic system of signification and signaling includes a way as to be able to show the user works as different as the movements of the body, face, and eyes [23]. Facial possible. The second function was to collect information expressions manifest the intentions of the subject based about the visitor. In particular, we were interested in on the context and depending on this there are facial ex- acquiring data relating to her demographic profile, de- pressions that differ substantially, also giving the listener gree of appreciation of the work displayed at that time, the possibility to understand the state of mind of her and resulting micro facial expressions. Specifically, par- interlocutor. In 1979 Paul Ekman and Wallace V. Friesen, ticipants were shown eight artworks and asked to rate based on the previously developed study by Swedish each of them on a five-point Likert scale. Meanwhile, anatomist Carl-Herman Hjortsjö [24], proposed the Fa- the participants were recorded through the webcam of cial Action Coding System (FACS) [23], an anatomically their device while viewing each artwork. Demographic accurate system to describe all visually distinguishable information was collected through a final questionnaire. facial movements. Specifically, the demographic data relating to the users who participated in the experimental trials are shown 2.1.1. Action Units (AUs) in Table 1. The participants were 73, almost equally dis- tributed between females and males, and aged mostly The FACS decoding system explores facial expressions between 21 and 29. Most participants had a high school by breaking them down into the smallest fundamental diploma and were mainly university students. Once the units, the action units (AUs), giving each one a meaning. dataset was collected, it was necessary to process the Ekman and Friesen cataloged 44 AUs describing changes recorded videos using facial recognition software. We in facial expressions and 14 AUs mapping changes in the employed two different software tools for this purpose: eye gaze direction and the head orientation. The AUs OpenFace3 , an opensource toolkit capable of perform- play a fundamental role in the recognition of emotions, ing action unit analysis, and iMotions4 , a proprietary movements, and attitudes, not only of the face but also software. of the body, allowing us to analyze the state of mind of the subject. The combination of the AUs enables us to map the four main emotions, namely, happiness, sadness, 4. Data Analysis anger, and fear [25]. Let us now analyze the results returned by the two analy- sis software. Table 2 shows the average values, standard 3. Data Collection deviations, as well as the minimum and maximum val- ues, calculated on the whole dataset. First of all, we can The research questions underlying the experimental anal- observe that the iMotions software returns more infor- ysis we performed are the following: is there a correlation mation than OpenFace and that the two software tools between the micro facial expressions of an observer and her degree of appreciation (i.e., rating) of an artwork? 1 https://www.raccoltadati.tk/ Is it possible to identify correlations shared by specific 2 https://lagallerianazionale.com/en/ categories of users? To answer these questions, it was 3 https://github.com/TadasBaltrusaitis/OpenFace 4 https://imotions.com/ Table 1 maximum value, and the deviation is very low. We can, Demographics of the 73 users involved in the experimental hence, conclude that most testers kept high their level analysis of attention during the virtual visit. Table 3 shows the value of Spearman’s correlation coefficient of the ratings Item Frequency assigned by the testers to the individual works and the Female 37 average score obtained by the features for each video. We Gender Male 36 Under 18 3 Table 3 18-20 5 Spearman’s correlation coefficient 21-29 40 Age 30-39 3 40-49 3 iMotions OpenFace 50-59 12 AU & Emotions Spearman’s Index Over 60 7 Inner Brow Raise -0.07 -0.06 Primary school 1 Outer Brow Raise -0.01 -0.05 8th grade diploma 9 Brow Lower -0.05 -0.06 Education High school diploma 41 Upper Lid Raise -0.05 University degree 19 Cheek Raise 0.00 -0.05 PhD 3 Lid Tighten -0.05 0.06 Unemployed 3 Nose Wrinkle -0.04 -0,04 Student 39 Upper Lip Raise -0.03 -0.04 Public employee 7 Lip Corner Puller 0.00 Profession Dimpler -0.02 -0.03 Private employee 14 Self-employed 7 Lip Corner Depressor -0.04 -0.06 Retired 3 Chin Raise 0.01 -0.07 Lip Stretch -0.09 -0.04 Lid Tighten -0.08 Mouth Open -0.01 0.00 Table 2 Jaw Drop -0.05 -0.02 Summary table of the output from the two software tools Blink -0.08 iMotions OpenFace Lip Suck -0.03 AU & Emotions Mean Std Min Max Mean Std Min Max Inner Brow Raise 5,099658 12,94021 0 80,29622 0,168434 0,141843 0,039858 1,462658 Lip Press -0.05 Brow Raise 3,565345 8,847247 0 55,49171 0,085252 0,061114 0,021103 0,478671 Brow Lower 5,334099 12,40427 0 76,77342 0,765825 0,739402 0 3,596304 Lip Pucker -0.06 Upper Lid Raiser 0,055565 0,031256 0,014244 0,245095 Cheek Raise 3,659209 10,67665 0 69,96562 0,390288 0,466435 0 2,387549 Eye Closure -0.17** Lid Tighten 0,93787 2,604525 0 23,44269 0,616453 0,719307 0 3,199208 Nose Wrinkle 0,973915 3,62885 0 44,94059 0,063658 0,054983 0,013989 0,350426 Eye Widen 0.03 Upper Lip Raise 1,135299 4,613869 0 44,57584 0,555492 0,527603 0 3,205763 Lip Corner Puller 0,397487 0,473547 0 2,572438 Smile -0.01 Dimpler 3,837253 7,598816 0 54,32411 0,570261 0,564813 0 2,724876 Lip Corner Depressor 1,766322 4,586096 0 41,22998 0,189903 0,220943 0,036511 1,946785 Smirk 0.04 Chin Raise 2,785176 5,867499 0 37,18328 0,407547 0,2544 0,080133 1,586465 Lip Stretch 2,535029 7,484421 0 61,21821 0,117077 0,11238 0,030426 1,131618 Engagement -0.04 Lip Tighten 0,93787 2,604525 0 23,44269 0,121904 0,123215 0,018549 0,929964 Mouth Open 6,867683 11,51858 0 66,08074 0,365305 0,331243 0,064533 2,580889 Attention -0.05 Jaw Drop 3,772275 6,797671 0 42,74697 0,36226 0,30048 0,0674 2,31789 Blink 0,169887 0,066811 0,041817 0,383651 Anger -0.05 Lip Suck 5,259716 9,547491 0 58,75693 Lip Press 2,926959 5,577136 0 31,28165 Sadness -0.13* Lip Pucker 2,870787 7,183146 0 46,96508 Eye Closure 1,966987 3,09202 0 30,51927 Disgust -0.02 Eye Widen 3,038526 7,873084 0 62,36883 Smile 7,651248 16,54695 0 82,14044 Joy -0.09 Smirk 2,030771 5,974433 0 62,60415 Engagement 15,29063 20,46839 0 88,82519 Surprise -0.07 Attention 93,17853 11,72724 15,89159 98,63756 Anger 0,473087 1,830745 0 21,59573 Fear -0.05 Sadness 0,869082 2,900364 0 28,76604 Disgust 1,297257 4,502729 0 61,42045 Contempt -0.07 Joy 5,829057 15,26311 0 83,61379 Surprise 1,364944 3,271783 0 31,10703 Fear 0,468503 1,842737 0 16,90147 Contempt 1,431101 5,146328 0 64,36057 can immediately notice a high correlation value between ratings and eye closure. The same thing happens for per- sometimes analyze the same micro expressions. The ceived sadness. The negative value of these correlations mean of the individual action units is often less than indicates that a high value of the feature corresponds to the standard deviation. At the same time, the minimum a low rating attributed to the work. We then verified if values differ highly from the maximum values. These there were any correlations shared by some categories of results, therefore, indicate the tendency of visitors to as- testers. More specifically, we grouped the data based on sume a neutral expression for most of the time except gender, the rating attributed to the artwork, and the num- in rare moments. The attention score, namely, the atten- ber of recognized artworks. Table 4 reports the values tion showed by the visitor while observing the artwork, returned by OpenFace. We note a positive correlation is noteworthy. The average value is very close to the value between the rating and the cheek raise action unit Table 4 sible to identify some correlation between facial micro Correlations between homogeneous groups in OpenFace expressions and the degree of appreciation of an object, Groups Male Female Low ratings High ratings Low frequency High frequency Few recognized Many recognized Low interest High interest specifically an artwork. It is also possible to identify cor- # Measurements 24 22 125 165 19 5 41 1 0 15 Inner Brow Raise Brow Raise -0.11 -0.05 -0.01 -0.04 -0.08 -0.08 0.01 0.06 -0.01 0.05 0.07 0.03 -0.06 -0.05 0.39 0.39 0 0 -0.07 -0.04 relations within some homogeneous groups of testers. Brow Lower -0.02 -0.09 -0.03 0.01 -0.06 0.00 -0.06 -0.39 0 -0.15 Upper Lid Raiser Cheek Raise Lid Tighten -0.11 -0.04 -0.05 0.02 0.15* 0.17* -0.11 -0.06 -0.05 -0.07 -0.04 0.01 0.05 0.05 0.04 -0.06 0.06 0.19 -0.06 0.06 0.12* 0.23 -0.49 0.71* 0 0 0 -0.04 -0.02 0.00 Our experimental analysis is very simplified and also Nose Wrinkle Upper Lip Raise Lip Corner Puller -0.06 -0.02 -0.05 -0.02 -0.09 0.05 -0.15 -0.19* -0.19* -0.06 -0.07 -0.01 -0.08 0.06 0.06 0.38* -0.16 -0.03 -0.05 -0.06 0.00 0.00 -0.05 0.10 0 0 0 0.02 -0.03 -0.01 suffers from numerous limitations. Among others, it is Dimpler Lip Corner Depressor Chin Raise -0.02 -0.10 -0.10 -0.03 0.00 -0.03 -0.17 -0.05 -0.13 -0.07 0.04 -0.05 0.10 0.01 -0.08 -0.06 0.18 0.20 -0.04 -0.06 -0.06 0.15 -0.28 -0.23 0 0 0 -0.09 -0.10 -0.09 evident as follows: Lip Stretch -0.09 0.03 0.00 0.00 0.06 0.10 -0.05 -0.15 0 -0.08 Lip Tighten -0.08 -0.05 -0.09 -0.01 -0.07 0.04 -0.10 -0.54 0 -0.04 Mouth Open Jaw Drop Blink -0.04 -0.06 -0.08 0.04 0.04 -0.08 -0.14 -0.03 -0.06 0.07 0.01 -0.09 0.03 0.03 -0.05 0.18 0.16 0.33* 0.00 -0.02 -0.08 0.13 -0.31 -0.28 0 0 0 -0.05 -0.05 -0.10 • it was performed in a specific domain, namely p < .0001 ‘****’; p < .001 ‘***’, p < .01 ‘**’, p < .05 ‘*’ that of cultural heritage; • the micro facial expressions were collected in re- sponse to a specific stimulus, that is, the vision related to women. The same thing happens for the dis- of an artwork; tension of the eyelids, both for women and for those who • the data was collected through a virtual and not have recognized few works. Finally, for those who as- live experimentation; signed a low rating, we found a negative correlation for • the sample of users was very limited; the lifting of the lips and their sinking. In Table 5, we • the sample of users was mostly made up of uni- can instead observe the correlation values calculated on versity students, so it was anything but heteroge- the results of iMotions. We can observe how eye closure neous. Table 5 A much more extensive and rigorous experimental anal- Correlations between homogeneous groups in iMotions ysis is therefore needed, including further categories Groups Male Female Low High Low High Few Many Low High of users, scenarios (e.g., [26, 27, 28]), and information ratings ratings frequency frequency recognized recognized interest interest # Measurements Brow Furrow 24 -0.04 22 -0.06 125 -0.05 165 0.04 19 0.00 5 0.11 41 -0.05 1 0.28 0 0 15 -0.08 (e.g., [29]). Only in this way we could indeed draw defini- tive conclusions on the existence of correlations between Brow Raise -0.02 0.00 -0.08 0.00 -0.06 0.12 -0.01 0.33 0 0.03 Engagement -0.12 0.04 -0.12 0.04 -0.03 -0.04 -0.04 0.00 0 -0.11 Lip Corner Depressor -0.07 0.01 0.05 0.03 -0.05 0.35* -0.06 -0.49 0 0.02 Smile -0.13 0.11 -0.19* 0.01 0.05 -0.07 0.04 -0.18 0 -0.11 Attention Inner Brow Raise Eye Closure 0.00 -0.13 -0.13 -0.10 0.01 -0.21** 0.15 0.06 -0.02 -0.15 -0.04 -0.07 -0.22** -0.09 -0.20* -0.14 0.17 0.19 -0.03 -0.06 -0.21*** -0.69 -0.67 -0.08 0 0 0 -0.03 -0.04 -0.09 micro facial expressions and categories of testers. Nose Wrinkle -0.06 0.01 -0.14 -0.03 -0.02 0.01 -0.01 0.08 0 -0.07 Upper Lip Raise -0.05 0.01 -0.12 -0.05 -0.01 0.08 -0.01 0.31 0 -0.04 Lip Suck -0.07 0.00 -0.09 0.02 -0.05 -0.02 -0.02 0.80* 0 -0.08 Lip Press -0.09 0.00 -0.09 -0.02 -0.04 0.03 -0.05 0.44 0 -0.06 References Mouth Open -0.05 0.03 -0.11 0.02 0.08 0.14 0.01 0.10 0 -0.11 Chin Raise -0.06 0.11 -0.08 0.06 -0.06 -0.04 0.00 0.28 0 0.02 Smirk -0.01 0.12 -0.13 0.06 0.06 -0.11 0.02 0.69 0 0.06 Lip Pucker 0.06 0.06 -0.13 0.04 0.03 -0.06 0.02 -0.05 0 0.07 Anger -0.05 -0.04 0.02 0.06 -0.01 0.16 -0.08 0.33 0 -0.08 Sadness -0.14* -0.13 -0.04 0.00 -0.19* 0.13 -0.15** -0.23 0 -0.06 Disgust 0.01 -0.03 -0.04 0.02 0.00 0.33* -0.05 -0.10 0 0.05 [1] X. Alameda-Pineda, E. Ricci, N. Sebe, Multimodal Joy -0.16* 0.00 -0.16 -0.02 -0.05 0.01 -0.05 -0.36 0 -0.17* Surprise -0.07 -0.07 -0.12 -0.02 -0.11 -0.21 -0.08 0.33 0 -0.01 Fear -0.05 -0.05 0.02 -0.07 -0.02 0.06 -0.12* -0.08 0 -0.05 Contempt -0.07 -0.06 0.02 0.06 -0.08 0.04 -0.12* 0.72* 0 0.01 Cheek Raise Dimpler Eye Widen -0.11 -0.09 0.04 0.11 0.05 0.03 -0.17 -0.12 -0.02 0.03 -0.04 -0.08 0.05 0.02 0.13 -0.05 -0.04 -0.35* 0.04 -0.02 -0.04 0.03 0.41 -0.08 0 0 0 -0.10 -0.07 0.00 behavior analysis in the wild: An introduction, in: Lid Tighten Lip Stretch -0.13 -0.11 0.02 -0.07 -0.07 -0.16 p < .0001 ‘****’; p < .001 ‘***’, p < .01 ‘**’, p < .05 ‘*’ -0.02 -0.12 -0.18* -0.03 0.27 -0.25 -0.01 -0.07 -0.39 -0.08 0 0 -0.04 -0.12 X. Alameda-Pineda, E. Ricci, N. Sebe (Eds.), Mul- timodal Behavior Analysis in the Wild, Computer is negatively correlated in the two groups. Also, sadness Vision and Pattern Recognition, Academic Press, is negatively correlated in the two groups. Joy is also 2019, pp. 1–8. negatively correlated. And, finally, fear and contempt are [2] B. T. Hung, L. M. Tien, Facial expression recognition negatively correlated for the same group of people who with cnn-lstm, in: R. Kumar, N. H. Quang, V. Ku- recognized a few works. These results, therefore, show mar Solanki, M. Cardona, P. K. Pattnaik (Eds.), Re- how all features are negatively correlated, thus express- search in Intelligent and Computing in Engineering, ing that a high score of a given variable corresponds to a Springer Singapore, Singapore, 2021, pp. 549–560. lower score. Further analyzes that we cannot report for [3] Y. Tian, T. Kanade, J. F. Cohn, Facial expression reasons of space show how visitors who did not like the recognition, in: S. Z. Li, A. K. Jain (Eds.), Hand- works expressed their low ratings more clearly. book of Face Recognition, Springer London, Lon- don, 2011, pp. 487–519. [4] C. Darwin, The expression of the emotions in man 5. Conclusions and Future Works and animals, London, UK: John Marry (1872). [5] E. Friesen, P. Ekman, The facial action coding sys- The ultimate goal of our research activities was to ver- tem: a technique for the measurement of facial ify whether facial micro expressions can be exploited to movement, Palo Alto 3 (1978). create interfaces that can adapt differently depending on [6] P. Ekman, K. R. Scherer, Handbook of methods in the characteristics of the active user. If so, it would be nonverbal behavior research, Cambridge University possible to foster cultural and social inclusion between Press, 1982. individuals from different backgrounds and belonging to [7] M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, different categories, including disadvantaged and at-risk I. Fasel, J. Movellan, Automatic recognition of fa- categories as well as vulnerable people. In particular, cial actions in spontaneous expressions, Journal of from the experimental results, it emerged how it is pos- Multimedia 1 (2006) 22–35. [8] H. Gunes, M. Piccardi, Automatic temporal segment dentlitteratur, 1969. detection and affect recognition from face and body [25] C. G. Kohler, T. Turner, N. M. Stolar, W. B. Bilker, display, IEEE Transactions on Systems, Man, and C. M. Brensinger, R. E. Gur, R. C. Gur, Differences Cybernetics, Part B (Cybernetics) 39 (2008) 64–84. in facial expressions of four universal emotions, [9] J. Lien, T. Kanade, J. Cohn, C. Li, Detection, tracking, Psychiatry Research 128 (2004) 235 – 244. and classification of action units in facial expression, [26] S. Caldarelli, D. F. Gurini, A. Micarelli, G. Sansonetti, Robotics and Autonomous Systems 31 (2000). A signal-based approach to news recommendation, [10] Y.-l. Tian, T. Kanade, J. F. Cohn, Recognizing action in: CEUR Workshop Proceedings, volume 1618, units for facial expression analysis, IEEE Trans. CEUR-WS.org, Aachen, Germany, 2016. Pattern Anal. Mach. Intell. 23 (2001) 97–115. [27] M. Onori, A. Micarelli, G. Sansonetti, A comparative [11] J. Carroll, J. Russell, Do facial expressions signal analysis of personality-based music recommender specific emotions? judging emotion from the face systems, in: CEUR Workshop Proceedings, volume in context., Journal of personality and social psy- 1680, CEUR-WS.org, Aachen, Germany, 2016. chology 70 2 (1996) 205–18. [28] D. Valeriani, G. Sansonetti, A. Micarelli, A compar- [12] J. Russell, Culture and the categorization of emo- ative analysis of state-of-the-art recommendation tions., Psychological bulletin 110 3 (1991) 426–50. techniques in the movie domain, Lecture Notes in [13] J. Russell, Is there universal recognition of emo- Computer Science 12252 LNCS (2020) 104–118. tion from facial expression? a review of the cross- [29] M. Saneiro, O. Santos, S. Salmeron-Majadas, J. Bot- cultural studies., Psychological bulletin 115 1 (1994). icario, Towards emotion detection in educational [14] G. Sansonetti, Point of interest recommendation scenarios from facial expressions and body move- based on social and linked open data, Personal and ments through multimodal approaches, The Scien- Ubiquitous Computing 23 (2019) 199–214. tific World Journal (2014). [15] H. A. M. Hassan, G. Sansonetti, F. Gasparetti, A. Mi- carelli, Semantic-based tag recommendation in sci- entific bookmarking systems, in: Proceedings of the 12th ACM Conference on Recommender Sys- tems, ACM, New York, NY, USA, 2018, pp. 465–469. [16] A. Fogli, G. Sansonetti, Exploiting semantics for context-aware itinerary recommendation, Personal and Ubiquitous Computing 23 (2019) 215–231. [17] M. Chang, G. D’Aniello, M. Gaeta, F. Orciuoli, D. Sampson, C. Simonelli, Building ontology-driven tutoring models for intelligent tutoring systems us- ing data mining, IEEE Access 8 (2020) 48151–48162. [18] G. D’Aniello, M. Gaeta, F. Orciuoli, G. Sansonetti, F. Sorgente, Knowledge-based smart city service system, Electronics 9 (2020). [19] G. Sansonetti, F. Gasparetti, A. Micarelli, F. Cena, C. Gena, Enhancing cultural recommendations through social and linked open data, User Modeling and User-Adapted Interaction 29 (2019) 121–159. [20] M. Mezzini, C. Limongelli, G. Sansonetti, C. De Medio, Tracking museum visitors through convolutional object detectors, in: Adjunct Publication of UMAP ’20, ACM, New York, NY, USA, 2020, p. 352–355. [21] R. L. Birdwhistell, Kinesics and context: Essays on body motion communication, University of Penn- sylvania press, 2010. [22] A. Mehrabian, M. Wiener, Decoding of inconsistent communications, Journal of personality and social psychology 6 (1967) 109—114. [23] P. Ekman, W. Friesen, Facial Action Coding System, Consulting Psychologists Press, 1978. [24] C. Hjortsjö, Man’s Face and Mimic Language, Stu-