The Appearance of the Avatar Can Enhance the Sense of Co-Presence During Virtual Interactions with Users

Abstract

Computer media, interactive interfaces and virtual environments are becoming more social and increasingly used for interpersonal interactions. Indeed, nowadays we can observe an increment in social interactions between users and computer agents (i.e. avatars). The inclusion of avatars within virtual environments is known to facilitate interaction and to improve users' sense of presence (being in there) and social presence (being with another) during these simulations. Therefore, understanding which characteristics of avatars' appearance increase users' involvement and social presence in a virtual world can be useful for theoretical and applied purposes. We tried to address this issue by considering individuals' reaction to an ancient evolutionary behavior: grinding their mouth and showing their teeth to express anger. Participants determined their comfortdistance from male and female avatars showing anger and happiness with open and closed mouth. Results showed that comfort-distance was larger with angry avatars with open mouth than all other conditions. Participants approached happy avatars at shorter distances with both open and closed mouths. Therefore, if we control the way avatars manifest emotions by paying attention to their adaptive evolutionary roots, we can improve the capacity of virtual systems to induce a sense of co-presence, the feeling of being in touch with another in the computergenerated world.

Keywords

• Emotions - Adaptive function- Avatars' appearance - Social pres- ence - IVR

Introduction

Technological advances shape our society and thus deeply influence human thinking and behavior [ 1 ]. An example of such transformative technology is, for instance, the fact that computer media, interactive interfaces and virtual environments are becoming more social and increasingly used for interpersonal interactions [ 2, 3 ].

Indeed, nowadays we can observe an increment in social interactions not only among users, but also between users and computer agents (i.e. avatars), and this is particularly true for advanced interactive media, including Immersive Virtual Reality (IVR) [ 3, 4 ].

IVR is a medium that is able to induce the experience of “presence” in a computergenerated world. Presence refers to the “sense of being there” or the “feeling of being in a world that exists outside the self” [ 5-7 ]. Therefore, the sense of presence is rooted on a paradoxical state of consciousness: people behave and feel as if they actually were in the virtual world even though they know there is nothing real [ 5, 8 ]. The inclusion of avatars (i.e. representations of human users) within IVR environments is known to facilitate interaction and to improve users' sense of involvement and realism of this simulations [ 3 ]. Indeed, if the anthropomorphism of avatars is sufficiently “natural”, we tend to attribute them feelings and thoughts as if they had a mind similar to our own mind (e.g. [ 9, 10, 5, 11 ]). In other word, we develop the so-called “social presence”, the sense of “being together with another” [ 2, 12 ]. Therefore, beyond the “sense of being in the virtual place” there is the experience of “co-presence”, or quite essentially, the “sense of another through the virtual medium” [ 2, 12 ]. This phenomenon can be explained by the notion of “derivative intentionality” proposed by Dennett [ 13, 14 ]: the attribution of goals, desires, rationality to a system or an entity which hence referred to as intentional. Therefore, understanding which characteristics of avatars’ appearance increase the social presence can be useful for theoretical and applied purposes (e.g., [ 3 ]).

In this study we address this issue by considering a normal social behavior that is rooted in an ancient evolutionary behavior: opening or closing the mouth when we express emotions [ 15-18 ]. Facial expressions are salient social cues that signal an individual’s emotional state, behavioral intentions or action requests [ 17, 19, 20 ]. Therefore, successful social exchanges require both the ability to detect the emotions of others and to adapt our behavior to theirs [ 21 ].

According to an adaptive perspective, several emotional expressions may express a direct or indirect evolutionary root [ 15, 17, 20 ]. For example, grinding mouth and showing teeth is a typical behavior in the animal world to express aggression and attack against someone [22, 15, 17, 20, 23; see also [ 24 ]. Here we wondered if this evolutionary characteristic of emotions is also reflected in the appearance of avatars, that is if we are more afraid of avatars that seem angry with their mouths open or closed. And what happens with avatars that look happy? In other words, we want to exploit the adaptive functions of human facial expressions to explore the degree of social and physical involvement during social interactions with avatars.

According to proxemics, individuals actively use the space and manipulate their environment in order to establish their preferred degree of closeness and involvement [ 25 ] deploying approach and avoidance behaviors during interpersonal encounters [ 26 ].

In proxemics, this interpersonal space (IPS) represents the optimal social distance from conspecifics thus, a good measure of the quality of social interactions [ 27, 28 ]. In the social psychology literature, a typical task to assess the size of interpersonal space is based on comfort-distance judgments provided through the ‘approach-distance’ paradigm: participants have to stop themselves at the point where they still feel comfortable with the proximity of the interactant.

Typically, this distance increases in uncomfortable/threatening situations and decreases in comfortable/safe situations [ 27-30 ]. As signal of salient environmental events, emotional facial expressions trigger approaching-avoiding reactions that reveal evolutionary adaptations rooted in basic survival mechanisms [ 22, 15, 17, 22, 31 ] even when interacting with virtual conspecifics in IVR environments (e.g., [ 32 ]).

However, to our knowledge, it is still unclear if whether the way the face expresses emotions, i.e. with the mouth open or closed, can influence this basic survival response and, through distance modulation, provide an index of involvement in social-presence. To this end, we devised an IVR study in which participants were asked to determine the comfort-distance while approached virtual avatars showing anger and happiness with open and closed mouth. We hypothesize that if we are more afraid of avatars who seem angry with their mouths open than closed, then we should expect a larger distance in the former. Moreover, since previous studies have shown gender differences in interpersonal distance [ 25, 28, 33, 34, 35 ] we also investigate this aspect. 2 2.1

Method

Participants

Thirty-two right-handed participants (16 males), aged 18-29 years (Mage= 23; SD= 3), were recruited in exchange of course credit. Participants had normal/corrected-to-normal vision. Nobody claimed discomfort or vertigo during the IVR experience and reported being aware of the experimental purpose. All participants gave their written consent to take part in the study. The study was in conformity with the local Ethics Committee requirements and the Declaration of Helsinki (World Medical Association, 2013). 2.2

Setting and apparatus

The experimental setting and the virtual scenario were similar to those of previous studies [ 32-34 ]. The IVR equipment was installed in a 5x4x3 m room of the Laboratory of Cognitive Science and Immersive Virtual Reality (CS-IVR, Dept. Psychology). The equipment included the 3-D Vizard Virtual Reality Software Toolkit 4.10 (Worldviz, LLC, USA) with the Oculus Rift head mounted display (HMD) having two OLED displays for stereoscopic depth (images = 1920 x 1080; 90° horizontally, 110° diagonally). The IVR system continuously tracked and recorded participant’s position through a marker on the HMD. Visual information was updated in real time.

Virtual scenario and virtual stimuli

The virtual room (3 x 2.4 x 3 m) consisted of green walls, white ceiling and grey floor with a white dashed line from the initial position of the participant to the end of the virtual room. A total of eight young avatars (four females) were selected among a colony of highly realistic avatars and were used for the present study (Vizard Complete Characters, WorldViz; USA).

The emotional expression of the face was obtained by modelling the virtual faces with 3DS Max (Autodesk) following the KDEF database [ 36 ]. The sample of avatars was selected on the basis of a previous pilot study [ 32 ] in which 14 participants rated, on a 9-point Likert scale, how much the faces presented on the PC appeared happy/unhappy, friendly/threatening, angry/peaceful, and annoying/quite. Following this evaluation, twelve avatars were selected whose facial expressions were: happy (two males and two females) angry (two males and two females) and neutral (two males two females).

The selected sample (Fig.1) represented male and female adults aged about thirty years wearing similar casual clothes and were perceived as representation of Italian people. Their height was 175 cm (males) and 165 cm (females). Their gaze was kept looking straight ahead throughout the trials [ 37 ].

Procedure

After presenting the IVR devices, participants were invited to wear the HMD and to explore the virtual scenario in order to familiarize with the IVR equipment and the environment. Through the HMD, participants could see the virtual stimuli fully immersed in the virtual scene. During this initial experience they had to describe their feelings of presence (e.g. participants were asked if they felt present in the virtual space, if they felt they were acting in the virtual space excluding the real external environment). After that, participants received instructions about the comfort-task and then guided to the starting position holding a key-press device in their dominant hand. The entire experimental flow comprised a four-trial training session to allow the participant to familiarize with the task. Next, the tasting phase began with a short presentation of the instructions (2 s) followed by a fixation cross (300 ms) then an avatar (i.e. a male/female avatar showing angry or happy facial expression) appeared. The instruction was: “Press the button as soon as the distance between you and the avatar makes you feel uncomfortable’’. During the experimental session, avatars remained motionless and participants walked towards them (0.5 ms-1) until they stopped and simultaneously pressed the button, after which returned to the starting position. After button press, virtual stimuli disappeared. Each virtual stimulus was randomly presented 4 times (total of trials: 32), the experimental session lasted about 10 minutes. Finally, participants evaluated their experience with the avatars. They reported they clearly identified their facial expressions as if they were ‘‘realistic persons’’. 4 5

Data Analysis

Results

In each trial, the participant-avatar distance (cm) was recorded. The mean distances were analyzed through a 2x2x2x2 mixed ANOVA with Participant's Gender as between factor and three within factors: Emotion (2 levels: happy, angry), Avatars’ sex (male, female) and Mouth (2 levels: open, closed). The Tukey HSD post-hoc test was used, and the magnitude of significant effects was expressed by partial eta-squared (η²p). A main effect of Emotion appeared (F(1,30)= 22.28, p < .0001, η²p= 0.43). Comfort distance was larger with angry than happy avatars. A main effect of avatar's gender also emerged (F(1,30)= 14.85, p < .001, η² p= 0.33), such that comfort-distance was larger with male than female avatars. Finally, a main effect of mouth was shown (F(1,30)= 16.54, p < .001, η² p= 0.35). Comfort-distance was larger when dealing with avatars with open than closed mouth. Emotion and mouth significantly interacted (F(1,30)= 8.504, p = 0.007, η²p= 0.22). The effect was due to the fact that participants preferred a larger distance from angry avatars with open mouth than all other conditions (at least p < .001). Moreover, no difference emerged with avatars expressing happiness with open or closed mouth p= 0.92). Participant's gender, avatar's gender and mouth significantly interacted (F(1,30)= 6.13, p = 0.02, η² p= 0.17). The post-hoc analysis showed that male participants preferred a shorter comfort-distance from female avatars with closed mouth than all other conditions (at least p < .01). For female participants, comfort-distance was larger from male avatars with open mouth than all other conditions (at least p < .01). Finally, emotion, avatar's gender and mouth significantly interacted (F(1,30)= 6.95, p = 0.01, η² p= 0.19) (Fig.2). In presence of happy avatars, distance was shorter from females with both open/closed mouth (no difference, P= 0.93) than males with both open/closed mouth (at least p < .05). In presence of angry avatars, distance was larger from males with open mouth than all other conditions (at least p < .001). With female angry avatars, distance was larger when they had open than closed mouth (p < .001). The inclusion of avatars within IVR environments is known to facilitate interaction and to improve users' sense of involvement and realism of this virtual experiences [ 3 ]. Therefore, understanding which characteristics of the appearance of avatars may increase the social presence can be useful for theoretical and applied purposes (e.g. [ 3 ]). To this end, we have explored the effect of an ancient evolutionary behavior: opening or closing the mouth for expressing emotions [ 15-18 ]. Here we wondered if the primitive (defensive) response to (negative) emotions may be affected by the way the avatar expresses emotions, i.e. with the mouth open or closed, and, through distance modulation, provide an index of involvement in social-presence. In line with literature, results showed that participants kept further from angry avatars than happy ones [ 32 ] and from male than female avatars [ 33, 34 ]. In addition, male participants reduced distance with female avatars with closed mouth, while female participants increased distance with male avatars with open mouth. Coherently with previous findings (e.g., [ 35, 38 ], women seem to be more focused on gaining information about the safety/harmful value of the social context than men. More interestingly for our purpose, the present findings indicate that all the participants reacted by increasing distance when approached male angry avatars with open than closed mouth. Notably, no difference emerged between open and closed mouth when expressing happiness, and this is true both for male and female participants Indeed, going further previous works (e.g. [ 38 ]), interested in clarify gender-related and emotion-related effects and the ecological validity of VR in proxemics, here we explored which physical characteristics of virtual conspecifics per se could have reinforced the sense of presence and co-presence during IVR interactions. In line with this, our results show that avatars' appearance can enhance the sense of physical and social presence during IVR experiences and that reactions in response to emotions can reflect the degree of involvement in virtual interactions. As in the animal world, participants took advantages from those facial expressions able to enhance survival (e.g. [ 24 ]) reacting with avoiding behaviors against angry avatars showing teeth, a signal of potential aggression and attack [ 22, 15, 17, 20, 23 ]. On the other hand, as results confirm, in presence of emotions that communicate positivity, sociality and safety, people respond with approaching behavior, no matter if the other smiles with open or closed mouth [ 15, 17, 20, 23, 24 ]. According to an evolutionary perspective, it is more important to avoid potentially noxious stimuli, such as predators or other aggressive conspecifics, than to approach positive stimuli, such as smiling conspecifics [ 32 ]. To conclude, if we control the way avatars manifest emotions by paying attention to their adaptive evolutionary roots, we can improve the capacity of virtual systems to induce a sense of co-presence, the feeling of being in touch with another in the computer-generated world.

References

1. Dolata , U. : The transformative capacity of new technologies a theory of sociotechnical change . Hoboken: Taylor and Francis (Routledge Advances in Sociology) ( 2013 ).

2. Biocca , F. , Burgoon , J. , Harms , C. , Stoner , M. : Criteria and scope conditions for a theory and measure of social presence . In: 4th Annual International Workshop on Presence ( 2001 ).

3. Nowak , K. : Defining and Differentiating Copresence, Social Presence and Presence as Transportation . Presence 2001 Conference ( 2001 ).

4. Riva , G. , Mantovani , F. , Capideville , C.S. , Preziosa , A. , Morganti , F. , Villani , D. , Gaggioli , A. , Botella , C. , Alcañiz , M. : Affective interactions using virtual reality: The link between presence and emotions . Cyberpsychology and Behavior ( 2007 ). https://doi.org/10.1089/cpb. 2006 . 9993 .

5. Slater , M. : Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments . Philosophical Transactions of the Royal Society B: Biological Sciences ( 2009 ). https://doi.org/10.1098/rstb. 2009 . 0138 .

6. Slater , M. , Wilbur , S.: A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments . Presence: Teleoperators and Virtual Environments ( 1997 ). https://doi.org/10.1162/pres. 1997 . 6 .6.603.

7. Steuer , J.: Defining Virtual Reality: Dimensions Determining Telepresence . Journal of Communication ( 1992 ). https://doi.org/10.1111/j.1460- 2466 . 1992 .tb00812.x.

8. Sanchez-Vives , M. V. , Slater , M. : From presence to consciousness through virtual reality ( 2005 ). https://doi.org/10.1038/nrn1651.

9. Nowak , K. , Biocca , F. : Understanding the influence of agency and anthropomorphism on copresence, social presence and physical presence with virtual humans . Teleoperators and Virtual Environments , MIT Pres. 481 - 494 ( 2001 ).

10. Nowak , K.L. , Biocca , F. : The Effect of the Agency and Anthropomorphism on users' Sense of Telepresence, Copresence, and Social Presence in Virtual Environments . In: Presence: Teleoperators and Virtual Environments ( 2003 ). https://doi.org/10.1162/105474603322761289.

11. Riva , G. , Galimberti , C. : Computer-mediated communication: Identity and social interaction in an electronic environment . Genetic, Social, and General Psychology Monographs ( 1998 ).

12. Biocca , F. , Harms , C. , Burgoon , J.K. : Toward a More Robust Theory and Measure of Social Presence: Review and Suggested Criteria . Presence: Teleoperators and Virtual Environments ( 2003 ). https://doi.org/10.1162/105474603322761270.

13. Dennett , D.C. : Brainstorms MIT Press ( 1978 ).

14. Dennett , D.C. : The International Stance Cambridgele ( 1987 ).

15. Darwin , C. : The expression of emotion in animals and man ( 1872 ).

16. Ekman , P. : Universals and cultural differences in facial expressions of emotion . Nebraska Symposium on Motivation ( 1971 ).

17. Ekman , P. : Facial Expressions . In: Handbook of Cognition and Emotion ( 1999 ). https://doi.org/10.1002/0470013494.ch16.

18. Izard , C.E. : Emotions and facial expressions: A perspective from Differential Emotions Theory . In: J. A. Russell , J. M. Fernández-Dols (ed.) Studies in emotion and social interaction, 2nd series. The psychology of facial expression , pp. 57 - 77 ( 1997 ). https://doi.org/https://doi.org/10.1017/CBO9780511659911.005.

19. Keltner , D. , Ekman , P. , Gonzaga , G.C. , Beer , J. , Scherer , K.R. , Johnstone , T. , Klasmeyer , G. , Snowdon , C.T. , Oatley , K. , Gabrielsson , A. , Juslin , P.N. , Reilly , J. , Seibert , L. : Expression of emotion . In: Handbook of affective sciences ( 2003 ).

20. Fridlund , A.J.: The new ethology of human facial expressions . In: J. A. Russell , J. M. Fernández-Dols (ed.) Studies in emotion and social interaction, 2nd series. The psychology of facial expression , pp. 103 - 129 ( 1997 ). https://doi.org/https://doi.org/10.1017/CBO9780511659911.007.

21. Ferri , F. , Stoianov , I.P. , Gianelli , C. , D'Amico , L. , Borghi , A.M. , Gallese , V. : When action meets emotions: How facial displays of emotion influence goal-related behavior . PLoS ONE ( 2010 ). https://doi.org/10.1371/journal.pone. 0013126 .

22. Damasio , A.R.: The Feeling of What Happens: Body and Emotion in the Making of Consciousness ( 1999 ).

23. Fridlund , A.J. : Human facial expression: An evolutionary view . San Diego CA: Academic Press ( 1994 ).

24. Schmidt , K.L. , Cohn , J.F. : Human facial expressions as adaptations: Evolutionary questions in facial expression research . Yearbook of Physical Anthropology ( 2001 ). https://doi.org/10.1002/ajpa.20001.

25. Aiello , J.R. : Human Spatial Behavior . In: Stokols, D. , Altman , I. (ed.) Handbook of environmental psychology . pp. 389 - 504 . John Wiley and Sons, New York ( 1987 ).

26. Argyle , M. , Dean , J. : Eye-Contact , Distance And Affiliation . Soins; la revue de référence infirmière ( 1965 ). https://doi.org/10.2307/2786027.

27. Spencer , R.E.: Book Reviews: The Hidden Dimension by Edward T. Hall . New York: Doubleday and Company, Inc., 1966 . Pp. xii + 193. Educational and Psychological Measurement ( 1966 ). https://doi.org/10.1177/001316446602600462.

28. Hayduk , L.A. : Personal space: Where we now stand . Psychological Bulletin ( 1983 ). https://doi.org/10.1037/ 0033 - 2909 . 94 .2.293.

29. Kennedy , D.P. , Gläscher , J. , Tyszka , J.M. , Adolphs , R.: Personal space regulation by the human amygdala . Nature Neuroscience ( 2009 ). https://doi.org/10.1038/nn.2381.

30. Tajadura-Jiménez , A. , Pantelidou , G. , Rebacz , P. , Västfjäll , D. , Tsakiris , M.: I-space: The effects of emotional valence and source of music on interpersonal distance . PLoS ONE ( 2011 ). https://doi.org/10.1371/journal.pone. 0026083 .

31. Graziano , M.S.A. , Cooke , D.F. : Parieto-frontal interactions, personal space, and defensive behavior . Neuropsychologia 44 ( 6 ): 845 - 859 ( 2006 ). https://doi.org/10.1016/j.neuropsychologia. 2005 . 09 .011.

32. Ruggiero , G. , Frassinetti , F. , Coello , Y. , Rapuano , M. , di Cola , A.S. , Iachini , T. : The effect of facial expressions on peripersonal and interpersonal spaces . Psychological Research , 81 ( 2017 ). https://doi.org/10.1007/s00426-016-0806-x.

33. Iachini , T. , Coello , Y. , Frassinetti , F. , Ruggiero , G.: Body space in social interactions: A comparison of reaching and comfort distance in immersive virtual reality . PLoS ONE , 9 ( 2014 ). https://doi.org/10.1371/journal.pone. 0111511 .

34. Iachini , T. , Coello , Y. , Frassinetti , F. , Senese , V.P. , Galante , F. , Ruggiero , G.: Peripersonal and interpersonal space in virtual and real environments: Effects of gender and age . Journal of Environmental Psychology , 45 , ( 2016 ). https://doi.org/10.1016/j.jenvp. 2016 . 01 .004.

35. Uzzell , D. , Horne , N. : The influence of biological sex, sexuality and gender role on interpersonal distance . British Journal of Social Psychology ( 2006 ). https://doi.org/10.1348/014466605X58384.

36. Lundqvist , D. , Flykt , A. , Ohman , A. : The Karolinska directed emotional faces (KDEF) ( 1998 ). https://doi.org/10.1017/S0048577299971664.

37. Bailenson , J.N. , Blascovich , J. , Beall , A.C. , Loomis , J.M.: Interpersonal distance in immersive virtual environments ( 2003 ). https://doi.org/10.1177/0146167203029007002.

38. Rapuano , M. , Sbordone , F.L. , Borrelli , L.O. , Ruggiero , G. , Iachini , T. : The Effect of Facial Expressions on Interpersonal Space: A Gender Study in Immersive Virtual Reality . In: Smart Innovation, Systems and Technologies (Online 2020 ). https://doi.org/10.1007/ 978 - 981-15-5093-5_ 40 .